Back to Glossary Index

Dagster Data Engineering Glossary:


Data Wrangling

Convert unstructured data into a structured format.

Data wrangling definition:

Data wrangling, also known as data munging or data preprocessing, in the context of modern data engineering, refers to the process of manually converting or mapping data from one raw form into another format that allows for more convenient consumption of the data with the help of semi-automated tools. This involves cleaning, normalizing, and transforming data to eliminate any corruption or inaccuracies, ensure consistency, and make it suitable for analysis or use in machine learning models. The ultimate goal of data munging is to improve the quality and reliability of data insights and decisions derived from the data.

Data wrangling is the process of cleaning, structuring, and enriching raw data into a desired format for better decision-making in less time. This process can involve a wide range of operations, including parsing text data, dealing with missing or inconsistent data, converting data types, or even more complex tasks such as integrating multiple data sources.

Additionally, the transformation of unstructured data (like text, images, videos) into a structured format (like a database or spreadsheet) is often a key part of an ETL (Extract, Transform, Load) process in data engineering.

In the context of specific types of unstructured data like text, this process may also be referred to as text mining or text analytics, which involves extracting valuable information from unstructured text data and transforming it into a structured format.

An example of data wrangling using Python and Pandas

In this example, we'll demonstrate how to clean and transform a CSV file using the pandas library.

Given the following input file:

date,category,value
2023-01-01,A,10
2023-01-02,B,15
2023-01-03,A,8
2023-01-04,C,12
2023-01-05,B,20
2023-01-06,C,16
2023-01-07,A,11
2023-01-08,B,18
2023-01-09,C,14
2023-01-10,A,9
2023-01-11,B,16
2023-01-12,C,13

We can use the following code to clean and transform our data:

import pandas as pd

# Load the CSV file into a DataFrame
data = pd.read_csv('data.csv')

# Display the initial state of the data
print('Initial data:')
print(data.head())

# Drop any rows with missing values
data = data.dropna()

# Convert a column to datetime format
data['date'] = pd.to_datetime(data['date'])

# Filter the data for a specific date range
start_date = pd.to_datetime('2023-01-01')
end_date = pd.to_datetime('2023-05-01')
filtered_data = data[(data['date'] >= start_date) & (data['date'] <= end_date)]

# Group the data by a categorical column and calculate the average of a numeric column
grouped_data = filtered_data.groupby('category')['value'].mean().reset_index()

# Sort the grouped data by the average value in descending order
sorted_data = grouped_data.sort_values(by='value', ascending=False)

# Reset the index of the sorted data
sorted_data.reset_index(drop=True, inplace=True)

# Display the final state of the data
print('Final data:')
print(sorted_data.head())

In this example, we assume that you have a CSV file named data.csv containing columns such as 'date', 'category', and 'value'. Here's a breakdown of the steps involved:

  1. We start by importing the pandas library, which provides powerful data manipulation tools.
  2. The CSV file is loaded into a DataFrame using the read_csv function from pandas.
  3. We print the initial state of the data using the head function to display the first few rows.
  4. The dropna method is used to remove any rows with missing values from the DataFrame.
  5. The 'date' column is converted to datetime format using the to_datetime function.
  6. We filter the data for a specific date range using boolean indexing.
  7. The data is grouped by the 'category' column, and the mean value of the 'value' column is calculated using the groupby and mean functions.
  8. The grouped data is sorted in descending order based on the average value using the sort_values method.
  9. We reset the index of the sorted data using the reset_index function.
  10. Finally, we print the final state of the data using the head function to display the sorted and transformed DataFrame.

This example demonstrates some common data wrangling operations such as dropping missing values, converting data types, filtering data based on conditions, grouping data, and sorting. These operations are commonly performed in modern data engineering pipelines to clean and prepare data for further analysis or processing.

Here is the output this example will produce:

Initial data:
         date category  value
0  2023-01-01        A     10
1  2023-01-02        B     15
2  2023-01-03        A      8
3  2023-01-04        C     12
4  2023-01-05        B     20

Final data:
  category  value
0        B  17.25
1        C  13.75
2        A   9.50

Other data engineering terms related to
Data Transformation:
Dagster Glossary code icon

Align

Aligning data can mean one of three things: aligning datasets, meeting business rules, or arranging data elements in memory.
An image representing the data engineering concept of 'Align'
Dagster Glossary code icon

Clean or Cleanse

Remove invalid or inconsistent data values, such as empty fields or outliers.
An image representing the data engineering concept of 'Clean or Cleanse'
Dagster Glossary code icon

Cluster

Group data points based on similarities or patterns to facilitate analysis and modeling.
An image representing the data engineering concept of 'Cluster'
Dagster Glossary code icon

Curate

Select, organize, and annotate data to make it more useful for analysis and modeling.
An image representing the data engineering concept of 'Curate'
Dagster Glossary code icon

Denoise

Remove noise or artifacts from data to improve its accuracy and quality.
An image representing the data engineering concept of 'Denoise'
Dagster Glossary code icon

Denormalize

Optimize data for faster read access by reducing the number of joins needed to retrieve related data.
An image representing the data engineering concept of 'Denormalize'
Dagster Glossary code icon

Derive

Extracting, transforming, and generating new data from existing datasets.
An image representing the data engineering concept of 'Derive'
Dagster Glossary code icon

Discretize

Transform continuous data into discrete categories or bins to simplify analysis.
An image representing the data engineering concept of 'Discretize'
Dagster Glossary code icon

ETL

Extract, transform, and load data between different systems.
An image representing the data engineering concept of 'ETL'
Dagster Glossary code icon

Encode

Convert categorical variables into numerical representations for ML algorithms.
An image representing the data engineering concept of 'Encode'
Dagster Glossary code icon

Filter

Extract a subset of data based on specific criteria or conditions.
An image representing the data engineering concept of 'Filter'
Dagster Glossary code icon

Fragment

Break data down into smaller chunks for storage and management purposes.
An image representing the data engineering concept of 'Fragment'
Dagster Glossary code icon

Homogenize

Make data uniform, consistent, and comparable.
An image representing the data engineering concept of 'Homogenize'
Dagster Glossary code icon

Impute

Fill in missing data values with estimated or imputed values to facilitate analysis.
An image representing the data engineering concept of 'Impute'
Dagster Glossary code icon

Linearize

Transforming the relationship between variables to make datasets approximately linear.
An image representing the data engineering concept of 'Linearize'

Munge

See 'wrangle'.
An image representing the data engineering concept of 'Munge'
Dagster Glossary code icon

Normalize

Standardize data values to facilitate comparison and analysis. Organize data into a consistent format.
Dagster Glossary code icon

Reduce

Convert a large set of data into a smaller, more manageable form without significant loss of information.
An image representing the data engineering concept of 'Reduce'
Dagster Glossary code icon

Reshape

Change the structure of data to better fit specific analysis or modeling requirements.
An image representing the data engineering concept of 'Reshape'
Dagster Glossary code icon

Serialize

Convert data into a linear format for efficient storage and processing.
An image representing the data engineering concept of 'Serialize'
Dagster Glossary code icon

Shred

Break down large datasets into smaller, more manageable pieces for easier processing and analysis.
Dagster Glossary code icon

Skew

An imbalance in the distribution or representation of data.
Dagster Glossary code icon

Split

Divide a dataset into training, validation, and testing sets for machine learning model training.
Dagster Glossary code icon

Standardize

Transform data to a common unit or format to facilitate comparison and analysis.
Dagster Glossary code icon

Tokenize

Convert data into tokens or smaller units to simplify analysis or processing.
An image representing the data engineering concept of 'Tokenize'

Transform

Convert data from one format or structure to another.
An image representing the data engineering concept of 'Transform'