Data exploration definition:
Data Exploration, also called exploratory data analysis (EDA) refers to manual steps you can take to understand the data, identify patterns, and gain insights. This might involve examining data distributions, summary statistics, correlations, and visualizations.
Example of data exploration in Python
Let's assume that you want to work with data related to the sales of a supermarket. We will use the Pandas, Matplotlib and Seaborn libraries to perform the exploratory data analysis.
Your supermarket orders data is saved in a file called supermarket_sales.csv structured as follows:
ID,City,Customer_type,Gender,Product_line,Unit_price,Quantity,Tax_5%,Total,Date,Time,Payment,COGS,Gross_margin_percentage,Rating
1,Yangon,Member,Female,Health and beauty,74.69,7,26.14,548.97,1/5/2019,13:08,Ewallet,522.83,4.761904762,9.1
2,Naypyitaw,Normal,Male,Electronic accessories,15.28,5,3.82,80.22,3/8/2019,10:29,Cash,76.4,4.761904762,9.6
3,Yangon,Normal,Female,Home and lifestyle,46.33,7,16.22,340.52,2/3/2019,13:23,Credit card,324.31,4.761904762,7.4
4,Yangon,Member,Male,Health and beauty,58.22,8,23.29,489.05,2/27/2019,20:33,Ewallet,465.76,4.761904762,8.4
5,Yangon,Normal,Male,Sports and travel,86.31,7,30.21,634.38,2/8/2019,10:37,Ewallet,604.17,4.761904762,5.3
To perform some initial data exploration, we will do the following:
- Load the required libraries.
- Load the data from a CSV file.
- Perform basic exploration of the data including viewing the first few rows, printing summary statistics, and checking the shape of the data.
- Check for missing values.
- Print the unique values in each of the categorical columns.
We will then use Seaborn to plot some graphs:
- Visualize the data distribution of numerical columns using histograms.
- Visualize the correlation matrix to understand the relationships between numerical variables.
- Generate a pair plot of the first five columns to visualize relationships and distribution of each column.
- Finally, create a box plot to see the relationship between a categorical variable and a numerical variable.
Please note that you need to have the necessary Python libraries installed in your Python environment to run this code.
# Import required libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# Load data from CSV file
df = pd.read_csv('supermarket_sales.csv')
# Print first 5 rows of the dataframe
print(df.head())
# Print the summary statistics
print(df.describe())
# Print the info about dataframe
print(df.info())
# Print the shape of the dataframe
print("The dataset contains ", df.shape[0], "rows and ", df.shape[1], "columns")
# Checking for missing values
print(df.isnull().sum())
# Check the unique values in categorical columns
for col in df.select_dtypes(include=['object']).columns:
print("Unique values in column: ", col)
print(df[col].unique())
# Histogram for the numerical columns
df.hist(bins=50, figsize=(20,15))
plt.show()
# Correlation matrix
corr_matrix = df.corr()
sns.heatmap(corr_matrix, annot=True)
plt.show()
# Pair plot for the first five columns
sns.pairplot(df.iloc[:, :5])
plt.show()
# Box plot for a categorical column against a numerical column
sns.boxplot(x='City', y='Total', data=df)
plt.show()
Let's look at the output
Running the code above will generate a number of outputs for us to look at. Let's walk through them in order:
# Print first 5 rows of the dataframe
print(df.head())
This first command sImply gives us the first five rows from our csv file, now turned into a Pandas Dataframe:
ID City Customer_type Gender Product_line Unit_price Quantity Tax_5% Total Date Time Payment COGS Gross_margin_percentage Rating
0 1 Yangon Member Female Health and beauty 74.69 7 26.14 548.97 1/5/2019 13:08 Ewallet 522.83 4.761905 9.1
1 2 Naypyitaw Normal Male Electronic accessories 15.28 5 3.82 80.22 3/8/2019 10:29 Cash 76.40 4.761905 9.6
2 3 Yangon Normal Female Home and lifestyle 46.33 7 16.22 340.52 2/3/2019 13:23 Credit card 324.31 4.761905 7.4
3 4 Yangon Member Male Health and beauty 58.22 8 23.29 489.05 2/27/2019 20:33 Ewallet 465.76 4.761905 8.4
4 5 Yangon Normal Male Sports and travel 86.31 7 30.21 634.38 2/8/2019 10:37 Ewallet 604.17 4.761905 5.3
Next, we look at the summary statistics...
# Print the summary statistics
print(df.describe())
...which in our case gives us:
ID Unit_price Quantity Tax_5% Total COGS Gross_margin_percentage Rating
count 5.000000 5.00000 5.000000 5.000000 5.000000 5.000000 5.000000 5.000000
mean 3.000000 56.16600 6.800000 19.936000 418.628000 398.694000 4.761905 7.960000
std 1.581139 27.49741 1.095445 10.354083 217.438528 207.083548 0.000000 1.700882
min 1.000000 15.28000 5.000000 3.820000 80.220000 76.400000 4.761905 5.300000
25% 2.000000 46.33000 7.000000 16.220000 340.520000 324.310000 4.761905 7.400000
50% 3.000000 58.22000 7.000000 23.290000 489.050000 465.760000 4.761905 8.400000
75% 4.000000 74.69000 7.000000 26.140000 548.970000 522.830000 4.761905 9.100000
max 5.000000 86.31000 8.000000 30.210000 634.380000 604.170000 4.761905 9.600000
Then we return the dataframe details and shape...
# Print the info about dataframe
print(df.info())
# Print the shape of the dataframe
print("The dataset contains ", df.shape[0], "rows and ", df.shape[1], "columns")
... which yields:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 15 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 ID 5 non-null int64
1 City 5 non-null object
2 Customer_type 5 non-null object
3 Gender 5 non-null object
4 Product_line 5 non-null object
5 Unit_price 5 non-null float64
6 Quantity 5 non-null int64
7 Tax_5% 5 non-null float64
8 Total 5 non-null float64
9 Date 5 non-null object
10 Time 5 non-null object
11 Payment 5 non-null object
12 COGS 5 non-null float64
13 Gross_margin_percentage 5 non-null float64
14 Rating 5 non-null float64
dtypes: float64(6), int64(2), object(7)
memory usage: 728.0+ bytes
None
The dataset contains 5 rows and 15 columns
We check for missing values with
# Checking for missing values
print(df.isnull().sum())
... which confirms that none of the columns have missing values:
ID 0
City 0
Customer_type 0
Gender 0
Product_line 0
Unit_price 0
Quantity 0
Tax_5% 0
Total 0
Date 0
Time 0
Payment 0
COGS 0
Gross_margin_percentage 0
Rating 0
dtype: int64
Next we check for unique values:
# Check the unique values in categorical columns
for col in df.select_dtypes(include=['object']).columns:
print("Unique values in column: ", col)
print(df[col].unique())
which gives us a convenient breakdown of the unique values in the dataset:
Unique values in column: City
['Yangon' 'Naypyitaw']
Unique values in column: Customer_type
['Member' 'Normal']
Unique values in column: Gender
['Female' 'Male']
Unique values in column: Product_line
['Health and beauty' 'Electronic accessories' 'Home and lifestyle'
'Sports and travel']
Unique values in column: Date
['1/5/2019' '3/8/2019' '2/3/2019' '2/27/2019' '2/8/2019']
Unique values in column: Time
['13:08' '10:29' '13:23' '20:33' '10:37']
Unique values in column: Payment
['Ewallet' 'Cash' 'Credit card']
Let's do some visualization
So far we have returned insights into the shape and statistical profile of our dataset. Next let's use Seaborn to create some histograms of our data.
About Seaborn
Seaborn is a Python data visualization library based on Matplotlib. It provides a high-level interface for creating informative and attractive statistical graphics.
Seaborn is great for exploratory data analysis (EDA) because it makes it easy to visualize complex patterns in data. It's especially good for visualizing statistical models, such as linear regression. However, Seaborn is a layer on top of Matplotlib, so it can be a bit more challenging to use for custom visualizations. For such cases, Matplotlib might be a better option.
Here are some of the features that Seaborn offers:
Built-in themes: Seaborn comes with a number of customized themes and a high-level interface for controlling the look of Matplotlib figures.
Visualizing univariate and bivariate distributions: Seaborn allows you to visualize distributions of one or two variables including histograms, density plots, and scatter plots.
Aggregated and conditional plots: Seaborn can create plots that show the relationship between variables with aggregation and conditioning (like boxplots, violin plots, and pair plots).
Heatmaps: Seaborn is great for creating heatmaps, which can be used for anything from correlation matrices to confusion matrices.
Integration with Pandas: Seaborn works well with Pandas dataframes and arrays that contain datasets. It also integrates closely with pandas data structures.
Multi-plot grids: Seaborn makes it easy to setup multi-plot grids for laying out complex visualizations.
The graphical output
Granted, our dataset is small (just five items) and simple so our data visualization might be underwhelming, but this is a basic demonstration and you can build from here. Feel free to generate a more complicated dataset to make these examples more interesting.
Visualize the data distribution of numerical columns using histograms.
Out first visualization only uses Matplotlib, and will take the numerical columns from our dataset and return a basic histogram for each:
# Histogram for the numerical columns
df.hist(bins=50, figsize=(20,15))
plt.show()
Which gives us this:
If you like, you can replace those two lines of code with something more interesting which will plot just one of the columns, say Ratings.
# Create a histogram of the 'Rating' column
df['Rating'].hist(bins=5, edgecolor='black')
plt.title('Distribution of Ratings')
plt.xlabel('Rating')
plt.ylabel('Frequency')
plt.show()
Our vizualisation now makes more sense:
Visualize the correlation matrix to understand the relationships between numerical variables.
Here we use Seaborn with sns.heatmap:
# Correlation matrix
corr_matrix = df.corr()
sns.heatmap(corr_matrix, annot=True)
plt.show()
Generate a pair plot of the first five columns to visualize relationships and distribution of each column.
# Pair plot for the first five columns
sns.pairplot(df.iloc[:, :5])
plt.show()
Create a box plot to see the relationship between a categorical variable (City) and a numerical variable (Total).
# Box plot for a categorical column against a numerical column
sns.boxplot(x='City', y='Total', data=df)
plt.show()
Again, the plot is very simple with just five rows of data, but you can explore from here.
Cosine Similarity
Extrapolate
Geospatial Analysis
Interpolate
Mine
Normality Testing
Profile
Sentiment Analysis
Time Series Analysis
Unstructured Data Analysis
