Python is one of the most popular programming languages in the world of data science. It provides a variety of tools and packages for data cleaning and preparation. In this post, you’ll find crucial and commonly used Python Snippets for Data Cleaning and Preparation in the fields of AI, ML, and data science.
Python Snippets for Data Cleaning and Preparation: Removing Duplicates
Duplicate data is one of the most common problems in any dataset. It can cause inaccuracies and lead to incorrect insights. Python provides a straightforward way to remove duplicates using the drop_duplicates() function. This function can be used on any pandas DataFrame, and it will remove all rows with duplicate values.
import pandas as pd
# create a dataframe with duplicate values
data = {'id': [1, 2, 3, 3, 4, 5], 'name': ['John', 'Bob', 'Alice', 'Alice', 'Dave', 'Jane']}
df = pd.DataFrame(data)
# remove duplicates
df.drop_duplicates(inplace=True)
print(df)
Output:
id name 0 1 John 1 2 Bob 2 3 Alice 4 4 Dave 5 5 Jane
Python Snippets for Data Cleaning and Preparation: Renaming Columns in a Dataset
Here is an example code to rename columns in a dataset using pandas library:
import pandas as pd
# create dummy data
data = {'col1': [1, 2, 3], 'col2': [4, 5, 6], 'col3': [7, 8, 9]}
df = pd.DataFrame(data)
# print the original dataframe
print('Original Dataframe:\n', df)
# rename columns
df = df.rename(columns={'col1': 'new_col1', 'col2': 'new_col2', 'col3': 'new_col3'})
# print the updated dataframe
print('Updated Dataframe:\n', df)Output:
Original Dataframe:
col1 col2 col3
0 1 4 7
1 2 5 8
2 3 6 9
Updated Dataframe:
new_col1 new_col2 new_col3
0 1 4 7
1 2 5 8
2 3 6 9This code creates a dummy dataframe with 3 columns named ‘col1’, ‘col2’, and ‘col3’. Then, the code uses the pandas “rename” method to rename the columns to ‘new_col1’, ‘new_col2’, and ‘new_col3’. Finally, the updated dataframe is printed to show the new column names.
Python Snippets for Data Cleaning and Preparation: Handling Missing Values
Missing values are another common problem in datasets. They can be caused by a variety of reasons such as incomplete data or errors in data collection. Python provides several functions to handle missing values.
Handling Missing Values: Removing Rows with Missing Values
In this approach, we drop all rows from a pandas DataFrame that contain at least one missing value using the dropna method with the inplace argument set to True.
import pandas as pd
# Create dummy data with missing values
data = {'A': [1, 2, None, 4, 5], 'B': [6, None, 8, 9, 10], 'C': [11, 12, 13, None, 15]}
df = pd.DataFrame(data)
# Drop rows with any missing values
df.dropna(inplace=True)
# Print the cleaned data
print(df)
Output:
A B C
0 1.0 6.0 11.0
3 4.0 9.0 14.0
Handling Missing Values: Removing Columns with Missing Values
In this approach, we remove all columns from a pandas DataFrame that contain at least one missing value using the dropna method with the axis argument set to 1 and the inplace argument set to True.
import pandas as pd
# Create dummy data with missing values
data = {'A': [1, 2, None, 4, 5], 'B': [6, None, 8, 9, 10], 'C': [11, 12, 13, None, 15]}
df = pd.DataFrame(data)
# Drop columns with any missing values
df.dropna(axis=1, inplace=True)
# Print the cleaned data
print(df)
Output:
A
0 1.0
1 2.0
2 NaN
3 4.0
4 5.0
Handling Missing Values: Filling Missing Values with a Constant
In this approach, we replace all missing values in a pandas DataFrame with a constant value using the fillna method with the value argument set to the desired constant and the inplace argument set to True.
import pandas as pd
# Create dummy data with missing values
data = {'A': [1, 2, None, 4, 5], 'B': [6, None, 8, 9, 10], 'C': [11, 12, 13, None, 15]}
df = pd.DataFrame(data)
# Fill missing values with a constant
df.fillna(value=0, inplace=True)
# Print the cleaned data
print(df)
Output:
A B C
0 1.0 6.0 11.0
1 2.0 0.0 12.0
2 0.0 8.0 13.0
3 4.0 9.0 0.0
4 5.0 10.0 15.0
Handling Missing Values: Filling Missing Values with Mean/Median
In this approach, we replace all missing values in a pandas DataFrame with the mean or median value of the corresponding column using the fillna method with the value argument set to the column-wise mean or median and the inplace argument set to True.
import pandas as pd
# Create dummy data with missing values
data = {'A': [1, 2, None, 4, 5], 'B': [6, None, 8, 9, 10], 'C': [11, 12, 13, None, 15]}
df = pd.DataFrame(data)
# Fill missing values with mean/median
df.fillna(df.mean(), inplace=True)
# Print the cleaned data
print(df)
Output:
A B C
0 1.0 6.0 11.0
1 2.0 8.0 12.0
2 2.5 8.0 13.0
3 4.0 9.0 12.75
4 5.0 10.0 15.0
Handling Missing Values: Interpolating Missing Values
You may choose to read this article on Interpolation vs Extrapolation: Common Methods with Python Code if you have no idea about interpolation & extrapolation. In this snippet, we use the interpolate method of pandas to fill in missing values using a linear interpolation method. The limit_direction argument is set to ‘forward’, which means that only missing values after a non-missing value will be filled in. If we wanted to fill in missing values both before and after non-missing values, we could set limit_direction to ‘both’.
import pandas as pd
# Create dummy data with missing values
data = {'A': [1, 2, None, 4, 5], 'B': [6, None, 8, 9, 10], 'C': [11, 12, 13, None, 15]}
df = pd.DataFrame(data)
# Interpolate missing values
df.interpolate(method='linear', limit_direction='forward', inplace=True)
# Print the cleaned data
print(df)
Output:
A B C
0 1.0 6.0 11.0
1 2.0 7.0 12.0
2 3.0 8.0 13.0
3 4.0 9.0 14.0
4 5.0 10.0 15.0
Python Snippets for Data Cleaning and Preparation: Changing Data Types
Sometimes, the data types of columns in a dataset may not be suitable for analysis. For example, a column with string values may need to be converted to numeric values. Python provides a simple way to change the data type of a column using the astype() function.
import pandas as pd
# create a dataframe with string values
data = {'id': [1, 2, 3, 4, 5], 'age': ['25', '30', '35', '40', '45']}
df = pd.DataFrame(data)
# change data type of age column from string to int
df['age'] = df['age'].astype(int)
print(df.dtypes)
Output:
id int64 age int32 dtype: object
Python Snippets for Data Cleaning and Preparation: Filtering rows based on a condition
Here is an example code to filter rows based on a condition using pandas library:
import pandas as pd
# create dummy data for the dataframe
data = {'id': [1, 2, 3, 4], 'name': ['John', 'Sarah', 'Mark', 'Anna'], 'age': [25, 30, 35, 20]}
df = pd.DataFrame(data)
# filter rows where age is greater than 30
filtered_df = df[df['age'] > 30]
# print the filtered dataframe
print(filtered_df)
Output:
id name age
2 3 Mark 35This code creates a dummy dataframe, df, with columns ‘id’, ‘name’, and ‘age’. Then, the code uses pandas indexing to filter the rows where the ‘age’ column is greater than 30. The resulting filtered dataframe is stored in the ‘filtered_df’ variable. Finally, the filtered dataframe is printed to show the filtered data. Note that multiple conditions can be combined using logical operators such as ‘&’ for ‘and’ and ‘|’ for ‘or’.
Python Snippets for Data Cleaning and Preparation: Sorting a dataset by a column
Here is an example code to sort a dataset by a column using pandas library:
import pandas as pd
# create dummy data for the dataframe
data = {'id': [1, 2, 3, 4], 'name': ['John', 'Sarah', 'Mark', 'Anna'], 'age': [25, 30, 35, 20]}
df = pd.DataFrame(data)
# sort the dataframe by age in descending order
sorted_df = df.sort_values('age', ascending=False)
# print the sorted dataframe
print(sorted_df)
Output:
id name age
2 3 Mark 35
1 2 Sarah 30
0 1 John 25
3 4 Anna 20This code creates a dummy dataframe, df, with columns ‘id’, ‘name’, and ‘age’. Then, the code uses the pandas “sort_values” method to sort the dataframe by the ‘age’ column in descending order. The resulting sorted dataframe is stored in the ‘sorted_df’ variable. Finally, the sorted dataframe is printed to show the sorted data. Note that the ‘ascending’ parameter can be set to False to sort in descending order, as shown in this example.
Python Snippets for Data Cleaning and Preparation: Grouping data by a column and aggregating values
Here is an example code to group data by a column and aggregate values using pandas library:
import pandas as pd
# create dummy data for the dataframe
data = {'name': ['John', 'Sarah', 'Mark', 'Anna', 'John'], 'age': [25, 30, 35, 20, 40], 'salary': [50000, 60000, 70000, 55000, 45000]}
df = pd.DataFrame(data)
# group the dataframe by name and aggregate age and salary columns
grouped_df = df.groupby('name').agg({'age': 'mean', 'salary': 'sum'})
# print the grouped dataframe
print(grouped_df)
Output:
age salary
name
Anna 20.0 55000
John 32.5 95000
Mark 35.0 70000
Sarah 30.0 60000This code creates a dummy dataframe, df, with columns ‘name’, ‘age’, and ‘salary’. Then, the code uses the pandas “groupby” method to group the data by the ‘name’ column. The ‘agg’ method is then used to aggregate the ‘age’ column by taking the mean, and the ‘salary’ column by taking the sum. The resulting grouped dataframe is stored in the ‘grouped_df’ variable. Finally, the grouped dataframe is printed to show the aggregated data. Note that other aggregation functions such as ‘min’, ‘max’, and ‘count’ can be used as well.
Python Snippets for Data Cleaning and Preparation: Applying a function to a column or dataframe
Here is an example code to apply a function to a column or a dataframe using pandas library:
import pandas as pd
# create dummy data for the dataframe
data = {'name': ['John', 'Sarah', 'Mark', 'Anna'], 'age': [25, 30, 35, 20]}
df = pd.DataFrame(data)
# define a function to add 5 years to age
def add_5_years(age):
return age + 5
# apply the function to the age column
df['age'] = df['age'].apply(add_5_years)
# print the updated dataframe
print(df)
Output:
name age
0 John 30
1 Sarah 35
2 Mark 40
3 Anna 25This code creates a dummy dataframe, df, with columns ‘name’ and ‘age’. Then, the code defines a function, ‘add_5_years’, to add 5 years to the input ‘age’ value. The ‘apply’ method is then used to apply this function to the ‘age’ column. The resulting updated dataframe is stored in the ‘df’ variable. Finally, the updated dataframe is printed to show the updated data. Note that the ‘apply’ method can be used to apply any function to a column or a dataframe.
Python Snippets for Data Cleaning and Preparation: Reshaping a dataframe from wide to long format or vice versa
Here is an example code to reshape a dataframe from wide to long format or vice versa using pandas library:
import pandas as pd
# create dummy data for the wide dataframe
wide_data = {'name': ['John', 'Sarah', 'Mark'], 'age_2020': [25, 30, 35], 'age_2021': [26, 31, 36], 'salary_2020': [50000, 60000, 70000], 'salary_2021': [55000, 65000, 75000]}
wide_df = pd.DataFrame(wide_data)
print('wide df:')
print(wide_df)
print('=========================')
# reshape the wide dataframe to long format
long_df = pd.melt(wide_df, id_vars=['name'], var_name='year_salary', value_name='value')
# print the long dataframe
print('long df:')
print(long_df)
print('=========================')
# reshape the long dataframe to wide format
wide_df_2 = long_df.pivot(index='name', columns='year_salary', values='value')
# print the wide dataframe
print('wide df 2:')
print(wide_df_2)
Output:
wide df:
name age_2020 age_2021 salary_2020 salary_2021
0 John 25 26 50000 55000
1 Sarah 30 31 60000 65000
2 Mark 35 36 70000 75000
=========================
long df:
name year_salary value
0 John age_2020 25
1 Sarah age_2020 30
2 Mark age_2020 35
3 John age_2021 26
4 Sarah age_2021 31
5 Mark age_2021 36
6 John salary_2020 50000
7 Sarah salary_2020 60000
8 Mark salary_2020 70000
9 John salary_2021 55000
10 Sarah salary_2021 65000
11 Mark salary_2021 75000
=========================
wide df 2:
year_salary age_2020 age_2021 salary_2020 salary_2021
name
John 25 26 50000 55000
Mark 35 36 70000 75000
Sarah 30 31 60000 65000This code creates a dummy dataframe, wide_df, in the wide format with columns ‘name’, ‘age_2020’, ‘age_2021’, ‘salary_2020’, and ‘salary_2021’. To reshape this wide dataframe to long format, the ‘melt’ method is used. The ‘id_vars’ parameter specifies the column(s) to use as identifier variable(s), while the ‘var_name’ parameter specifies the name of the new column to store the column names in, and the ‘value_name’ parameter specifies the name of the new column to store the values in. The resulting long dataframe is stored in the ‘long_df’ variable. Finally, the long dataframe is printed to show the long format.
To reshape the long dataframe back to the wide format, the ‘pivot’ method is used. The ‘index’ parameter specifies the column(s) to use as index, the ‘columns’ parameter specifies the column to pivot, and the ‘values’ parameter specifies the values to use for the new columns. The resulting wide dataframe is stored in the ‘wide_df_2’ variable. Finally, the wide dataframe is printed to show the wide format.
Python Snippets for Data Cleaning and Preparation: Merging datasets based on a common column
Here is an example code to merge datasets based on a common column using pandas library:
import pandas as pd
# create dummy data for the first dataframe
data1 = {'id': [1, 2, 3], 'name': ['John', 'Sarah', 'Mark']}
df1 = pd.DataFrame(data1)
# create dummy data for the second dataframe
data2 = {'id': [1, 2, 4], 'age': [25, 30, 35]}
df2 = pd.DataFrame(data2)
# merge the two dataframes based on the common 'id' column
merged_df = pd.merge(df1, df2, on='id', how='inner')
# print the merged dataframe
print(merged_df)
Output:
id name age
0 1 John 25
1 2 Sarah 30This code creates two dummy dataframes, df1 and df2, with a common column ‘id’. Then, the code uses the pandas “merge” method to merge the dataframes based on the ‘id’ column, with an inner join. Finally, the merged dataframe is printed to show the merged data. Note that there are different types of joins available, such as inner, outer, left, and right join, which can be specified using the ‘how’ parameter of the merge method.
Python Snippets for Data Cleaning and Preparation: Removing Outliers
Outliers are data points that are significantly different from the other data points in a dataset. They can skew the results of analysis and lead to incorrect insights. Python provides several ways to remove outliers, such as using the Z-score or IQR (Interquartile Range) method.
Here is an example of removing outliers using the Z-score method:
import pandas as pd
import numpy as np
# create a dataframe with outliers
data = {'id': [1, 2, 3, 4, 5], 'age': [25, 30, 35, 40, 100]}
df = pd.DataFrame(data)
# calculate Z-score for each data point
z_scores = np.abs((df['age'] - df['age'].mean()) / df['age'].std())
# remove outliers where the Z-score is greater than 3
df = df[z_scores < 3]
print(df)
Output:
id age 0 1 25 1 2 30 2 3 35 3 4 40
Python Snippets for Data Cleaning and Preparation: Handling Text Data
Text data is another common type of data that needs to be cleaned and prepared. Python provides several functions and packages for handling text data, such as the re package for regular expressions and the nltk package for natural language processing.
Here is an example of using the re package to remove punctuation from a text string:
import re
text = "Hello, World! How are you doing today?"
clean_text = re.sub(r'[^\w\s]', '', text)
print(clean_text)
Output:
Hello World How are you doing today
Python Snippets for Data Cleaning and Preparation: Splitting a dataset into training and testing sets for machine learning models
Here is an example code to split a dataset into training and testing sets for machine learning models using the scikit-learn library:
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
# load the iris dataset
iris = load_iris()
# split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3, random_state=42)
# print the shape of the training and testing sets
print('Shape of X_train:', X_train.shape)
print('Shape of y_train:', y_train.shape)
print('Shape of X_test:', X_test.shape)
print('Shape of y_test:', y_test.shape)Output:
Shape of X_train: (105, 4)
Shape of y_train: (105,)
Shape of X_test: (45, 4)
Shape of y_test: (45,)This code loads the iris dataset using the scikit-learn “load_iris” function. Then, the code uses the “train_test_split” function to split the dataset into training and testing sets with a test size of 30% and a random state of 42 for reproducibility. Finally, the code prints the shape of the training and testing sets to verify that the split was successful. Note that the “train_test_split” function can be used for any dataset, not just the iris dataset used in this example.
Conclusion
In conclusion, these five essential Python snippets can help you clean and prepare your data efficiently. By using these functions and packages, you can save time and ensure the accuracy of your analysis.
I highly recommend checking out this incredibly informative and engaging professional certificate Training by Google on Coursera:
Google Advanced Data Analytics Professional Certificate
There are 7 Courses in this Professional Certificate that can also be taken separately.
- Foundations of Data Science: Approx. 21 hours to complete. SKILLS YOU WILL GAIN: Sharing Insights With Stakeholders, Effective Written Communication, Asking Effective Questions, Cross-Functional Team Dynamics, and Project Management.
- Get Started with Python: Approx. 25 hours to complete. SKILLS YOU WILL GAIN: Using Comments to Enhance Code Readability, Python Programming, Jupyter Notebook, Data Visualization (DataViz), and Coding.
- Go Beyond the Numbers: Translate Data into Insights: Approx. 28 hours to complete. SKILLS YOU WILL GAIN: Python Programming, Tableau Software, Data Visualization (DataViz), Effective Communication, and Exploratory Data Analysis.
- The Power of Statistics: Approx. 33 hours to complete. SKILLS YOU WILL GAIN: Statistical Analysis, Python Programming, Effective Communication, Statistical Hypothesis Testing, and Probability Distribution.
- Regression Analysis: Simplify Complex Data Relationships: Approx. 28 hours to complete. SKILLS YOU WILL GAIN: Predictive Modelling, Statistical Analysis, Python Programming, Effective Communication, and regression modeling.
- The Nuts and Bolts of Machine Learning: Approx. 33 hours to complete. SKILLS YOU WILL GAIN: Predictive Modelling, Machine Learning, Python Programming, Stack Overflow, and Effective Communication.
- Google Advanced Data Analytics Capstone: Approx. 9 hours to complete. SKILLS YOU WILL GAIN: Executive Summaries, Machine Learning, Python Programming, Technical Interview Preparation, and Data Analysis.
It could be the perfect way to take your skills to the next level! When it comes to investing, there’s no better investment than investing in yourself and your education. Don’t hesitate – go ahead and take the leap. The benefits of learning and self-improvement are immeasurable.
You may also like:
- Top 5 Python programs you must know
- Python Snippets for Outliers Treatment: Essential Snippets for Data Cleaning and Preparation
Check out the table of contents for Product Management and Data Science to explore the topics. Curious about how product managers can utilize Bhagwad Gita’s principles to tackle difficulties? Give this super short book a shot. This will certainly support my work. After all, thanks a ton for visiting this website.