Data cleaning is an important part of data manipulation and analysis. We need to clean data with any null values, unknown characters, etc. Data cleaning is a time taking process which cannot be neglected because when we are preparing data for the machine learning model the data should be cleaned otherwise we won’t be able to generate useful insights. Or predictions.
We can apply different functions on the pandas dataframe which can help us in cleaning the data which in turn cleans the data, remove junk values, etc. But before that, we need to perform data analysis and know what all we need to do, what are the junk values, what are the datatypes of different columns in order to perform different operations for different datatypes. But what if we can automate this cleaning process? It can save a lot of time.
Datacleaner is an open-source python library which is used for automating the process of data cleaning. It is built using Pandas Dataframe and scikit-learn data preprocessing features. The contributors are actively updating it with new features. Some of the current features are:
- Dropping columns with null values
- Replacing null values with a mean(numerical data) and median(categorical data)
- Encoding non-numerical values with numerical equivalents.
In this article, we will see how datacleaner automates the process of data cleaning to save time and effort.
Implementation:
We will start by installing datacleaner using pip install datacleaner.
- Importing required libraries
We will be loading a dataset using pandas so we need to import pandas and for data cleaning, we will import autoclean function from datacleaner.
from datacleaner import autoclean
import pandas as pd
- Loading the required dataset
The dataset we are using in this article is a car design dataset that contains different attributes like ‘price’, ‘make’, ‘length’, etc. of different automobile companies. In this data, we will see that there are some junk values and some data is missing.
df = pd.read_csv('car_design.csv')
df.shape # Shape of the dataset
df.isnull().sum() #Checking Null Values
Here we can see that most of the columns contain null values. Now let us see the dataset.
print(df)
Here we can see that other than null values the data also contains some junk values as ‘?’. Now let us use autoclean and clean this data in just a single line of code.
clean_df = autoclean(df)
clean_df.shape
The shape remains the same as we have not dropped any column. Now let us see the null values.
It replaced all the null values with mean and median respectively. Now let us see what happened to junk values.
print(clean_df)
Here we can see that it also replaced all the junk values with the mean and median of that column respectively.
Conclusion:
In this article, we saw how we can clean data using data cleaner in just a single line of code. Autoclean removed all the junk values, missing values and cleaned the data so that it can be further used for machine learning models.
