Data Curation | Dagster Glossary

Back to Glossary Index

Data Curation

Select, organize, and annotate data to make it more useful for analysis and modeling.

Data curation definition:

Data curation refers to the process of organizing, maintaining, and preserving data to ensure its accuracy, consistency, and reliability. The goal of data curation is to maintain high-quality, trusted data throughout its lifecycle, from initial collection to long-term storage and retrieval. Many data curation practices are manual and require an understanding of the context in which the data was collected, or is being used.

Data curation best practices:

By implementing best practices and using the right tools, you can ensure that your organization’s data is well-maintained and usable throughout your data processes. Some best practices for data curation in data pipelines include:

  • Establishing clear data governance policies: This involves defining ownership, access rights, and security protocols for data. It helps to ensure that data is well-protected and only accessed by authorized personnel.
  • Documenting data lineage: Documenting the origin and processing history of data helps to ensure its traceability and reproducibility. It makes it easier to understand the context and meaning of data and helps to avoid errors and inconsistencies.
  • Implementing version control: Version control allows data engineers to keep track of changes made to data and helps to avoid accidental data loss or corruption. It ensures a reliable history of changes made to data over time.
  • Automating data validation: Data validation involves checking data for accuracy, completeness, and consistency. Automating this process helps to ensure that data is constantly checked and validated before being used in downstream processes.
  • Using data profiling tools: Data profiling tools help to identify patterns, anomalies, and errors in data. They make it easier to identify and correct data quality issues before they cause downstream problems.

Python libraries for data curation

In Python, there are several libraries and tools that can be used for data curation, such as:

  • Great Expectations: A library for automated data validation. It allows data engineers to define expectations about the structure and content of data and validate them automatically.
  • DVC: A version control system for data science and machine learning projects. It allows data scientists to track changes to data files and collaborate with team members.
  • DataProfiler: A library for data profiling and analysis. It allows data engineers to understand the structure and quality of data and identify potential issues.

An example of data curation in Python

Let's look at a basic example script that does the following:

  1. Downloads a public dataset (We will use the well-known Iris dataset)
  2. Loads the dataset into a DataFrame
  3. Removes any duplicates or rows with missing data
  4. Normalizes numeric data
  5. Splits the data into a training set and a test set
  6. Saves the cleaned, curated data to file

Here's the script. Please note that you must have the necessary Python libraries installed in your Python environment to run the code examples below.

import pandas as pd
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler

# Step 1: Download the dataset
iris = datasets.load_iris()
iris_df = pd.DataFrame(data=iris.data, columns=iris.feature_names)

# Step 2: Load the dataset into a DataFrame
iris_df['target'] = iris.target

# Step 3: Remove any duplicates or rows with missing data
iris_df.drop_duplicates(inplace=True)
iris_df.dropna(inplace=True)

# Step 4: Normalize numeric data
scaler = MinMaxScaler()
iris_df[iris.feature_names] = scaler.fit_transform(iris_df[iris.feature_names])

# Step 5: Split the data into a training set and a test set
train_df, test_df = train_test_split(iris_df, test_size=0.2, random_state=42)

# Step 6: Save the cleaned, curated data to disk
train_df.to_csv('train.csv', index=False)
test_df.to_csv('test.csv', index=False)

The output to train.csv and test.csv will look like this:

sepal length (cm),sepal width (cm),petal length (cm),petal width (cm),target
0.08333333333333326,0.6666666666666667,0.0,0.04166666666666667,0
0.38888888888888884,1.0,0.0847457627118644,0.125,0
0.6666666666666667,0.45833333333333326,0.576271186440678,0.5416666666666667,1
0.13888888888888884,0.5833333333333333,0.1016949152542373,0.04166666666666667,0
[...]

This script does a very basic job of data curation. A typical curation step would involve more sophisticated cleaning (like handling outliers, correcting mislabeled data, or dealing with unbalanced datasets) or data transformation (like feature engineering or encoding categorical variables). But the basic process will look similar: load the data, clean it, transform it, and save the curated data for later use.


Other data engineering terms related to
Data Transformation:
Dagster Glossary code icon

Align

Aligning data can mean one of three things: aligning datasets, meeting business rules, or arranging data elements in memory.
An image representing the data engineering concept of 'Align'
Dagster Glossary code icon

Clean or Cleanse

Remove invalid or inconsistent data values, such as empty fields or outliers.
An image representing the data engineering concept of 'Clean or Cleanse'
Dagster Glossary code icon

Cluster

Group data points based on similarities or patterns to facilitate analysis and modeling.
An image representing the data engineering concept of 'Cluster'
Dagster Glossary code icon

Denoise

Remove noise or artifacts from data to improve its accuracy and quality.
An image representing the data engineering concept of 'Denoise'
Dagster Glossary code icon

Denormalize

Optimize data for faster read access by reducing the number of joins needed to retrieve related data.
An image representing the data engineering concept of 'Denormalize'
Dagster Glossary code icon

Derive

Extracting, transforming, and generating new data from existing datasets.
An image representing the data engineering concept of 'Derive'
Dagster Glossary code icon

Discretize

Transform continuous data into discrete categories or bins to simplify analysis.
An image representing the data engineering concept of 'Discretize'
Dagster Glossary code icon

ETL

Extract, transform, and load data between different systems.
An image representing the data engineering concept of 'ETL'
Dagster Glossary code icon

Encode

Convert categorical variables into numerical representations for ML algorithms.
An image representing the data engineering concept of 'Encode'
Dagster Glossary code icon

Filter

Extract a subset of data based on specific criteria or conditions.
An image representing the data engineering concept of 'Filter'
Dagster Glossary code icon

Fragment

Break data down into smaller chunks for storage and management purposes.
An image representing the data engineering concept of 'Fragment'
Dagster Glossary code icon

Homogenize

Make data uniform, consistent, and comparable.
An image representing the data engineering concept of 'Homogenize'
Dagster Glossary code icon

Impute

Fill in missing data values with estimated or imputed values to facilitate analysis.
An image representing the data engineering concept of 'Impute'
Dagster Glossary code icon

Linearize

Transforming the relationship between variables to make datasets approximately linear.
An image representing the data engineering concept of 'Linearize'

Munge

See 'wrangle'.
An image representing the data engineering concept of 'Munge'
Dagster Glossary code icon

Normalize

Standardize data values to facilitate comparison and analysis. Organize data into a consistent format.
Dagster Glossary code icon

Reduce

Convert a large set of data into a smaller, more manageable form without significant loss of information.
An image representing the data engineering concept of 'Reduce'
Dagster Glossary code icon

Reshape

Change the structure of data to better fit specific analysis or modeling requirements.
An image representing the data engineering concept of 'Reshape'
Dagster Glossary code icon

Serialize

Convert data into a linear format for efficient storage and processing.
An image representing the data engineering concept of 'Serialize'
Dagster Glossary code icon

Shred

Break down large datasets into smaller, more manageable pieces for easier processing and analysis.
Dagster Glossary code icon

Skew

An imbalance in the distribution or representation of data.
Dagster Glossary code icon

Split

Divide a dataset into training, validation, and testing sets for machine learning model training.
Dagster Glossary code icon

Standardize

Transform data to a common unit or format to facilitate comparison and analysis.
Dagster Glossary code icon

Tokenize

Convert data into tokens or smaller units to simplify analysis or processing.
An image representing the data engineering concept of 'Tokenize'

Transform

Convert data from one format or structure to another.
An image representing the data engineering concept of 'Transform'
Dagster Glossary code icon

Wrangle

Convert unstructured data into a structured format.
An image representing the data engineering concept of 'Wrangle'