Back to Glossary Index

Discretize

Transform continuous data into discrete categories or bins to simplify analysis.

Data discretization definition:

Discretization is the process of converting continuous data into a set of discrete intervals or categories. This technique can be used for data reduction, simplification, or to make the data more suitable for analysis and it typically applied to very large datasets.

Data discretization examples using Python

Please note that you need to have the necessary Python libraries installed in your Python environment to run the code samples below.

using pandas cut()

One practical example of discretization in Python is using the pandas cut() function. The cut() function allows you to specify the number of bins you want to use and the range of the data, and it returns a new column with the values binned into those intervals. For example:

import pandas as pd

# create a sample dataset
data = pd.DataFrame({'values': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]})

# bin the values into three intervals
data['bins'] = pd.cut(data['values'], bins=3, labels=['Low', 'Medium', 'High'])

# view the binned data
print(data)

This will create a new column called 'bins' with the values binned into three intervals, 'Low', 'Medium', and 'High'. The output will be:

   values    bins
0       1     Low
1       2     Low
2       3  Medium
3       4  Medium
4       5  Medium
5       6    High
6       7    High
7       8    High
8       9    High
9      10    High

Another example of using using the pandas cut() function, if we have a DataFrame df containing a column ages , we can create bins and labels as follows:

import pandas as pd

# create a sample dataset
data = {'ages': [21, 32, 45, 18, 52, 28, 38, 50]}
df = pd.DataFrame(data)

bins = [0, 18, 35, 50, float('inf')]
labels = ['child', 'young adult', 'middle-aged', 'senior']

df['age_group'] = pd.cut(df['ages'], bins=bins, labels=labels)

print(df)

This will create a new column age_group in df containing the categorical labels based on the binning and print out the result:

   ages    age_group
0    21  young adult
1    32  young adult
2    45  middle-aged
3    18        child
4    52       senior
5    28  young adult
6    38  middle-aged
7    50  middle-aged

using scikit-learn

Another example of discretization in Python is using the KBinsDiscretizer class from the scikit-learn library. This class allows you to specify the number of bins, the strategy for dividing the data, and whether to encode the intervals as integers or one-hot vectors. For example:

from sklearn.preprocessing import KBinsDiscretizer

# create a sample dataset
data = [[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]]

# bin the values into three intervals
discretizer = KBinsDiscretizer(n_bins=3, encode='ordinal', strategy='uniform')
data_bins = discretizer.fit_transform(data)

# view the binned data
print(data_bins)

This will create a new array with the values binned into three intervals, and will yield the following:

[[0.]
 [0.]
 [1.]
 [1.]
 [1.]
 [2.]
 [2.]
 [2.]
 [2.]
 [2.]]

In this case, the encode parameter is set to ordinal, which means that the intervals are represented as integers. If it were set to onehot, the intervals would be represented as one-hot vectors.


Other data engineering terms related to
Data Transformation:

Align

Aligning data can mean one of three things: aligning datasets, meeting business rules or arranging data elements in memory.

Big Data Processing

Process large volumes of data in parallel and distributed computing environments to improve performance.

Clean or Cleanse

Remove invalid or inconsistent data values, such as empty fields or outliers.

Cluster

Group data points based on similarities or patterns to facilitate analysis and modeling.

Denoising

Remove noise or artifacts from data to improve its accuracy and quality.

Denormalize

Optimize data for faster read access by reducing the number of joins needed to retrieve related data.

ETL

Extract, transform, and load data between different systems.

Filter

Extract a subset of data based on specific criteria or conditions.

Fragment

Convert data into a linear format for efficient storage and processing.

Impute

Fill in missing data values with estimated or imputed values to facilitate analysis.

Munge

See 'wrangle'.

Normalize

Standardize data values to facilitate comparison and analysis. organize data into a consistent format.

Reduce

Convert a large set of data into a smaller, more manageable form without significant loss of information.

Reshape

Change the structure of data to better fit specific analysis or modeling requirements.

Serialize

Convert data into a linear format for efficient storage and processing.

Shred

Break down large datasets into smaller, more manageable pieces for easier processing and analysis.

Skew

An imbalance in the distribution or representation of data.

Standardize

Transform data to a common unit or format to facilitate comparison and analysis.

Tokenize

Convert data into tokens or smaller units to simplify analysis or processing.

Transform

Convert data from one format or structure to another.

Wrangle

Convert unstructured data into a structured format.