Back to Glossary Index


Select, organize and annotate data to make it more useful for analysis and modeling.

Data curation definition:

Data curation refers to the process of organizing, maintaining, and preserving data to ensure its accuracy, consistency, and reliability. The goal of data curation is to maintain high-quality, trusted data throughout its lifecycle, from initial collection to long-term storage and retrieval. Many data curation practices are manual and require an understanding of the context in which the data was collected, or is being used.

Data curation best practices:

By implementing best practices and using the right tools, you can ensure that your organization’s data is well-maintained and usable throughout your data processes. Some best practices for data curation in data pipelines include:

  • Establishing clear data governance policies: This involves defining ownership, access rights, and security protocols for data. It helps to ensure that data is well-protected and only accessed by authorized personnel.
  • Documenting data lineage: Documenting the origin and processing history of data helps to ensure its traceability and reproducibility. It makes it easier to understand the context and meaning of data and helps to avoid errors and inconsistencies.
  • Implementing version control: Version control allows data engineers to keep track of changes made to data and helps to avoid accidental data loss or corruption. It ensures that there is a reliable history of changes made to data over time.
  • Automating data validation: Data validation involves checking data for accuracy, completeness, and consistency. Automating this process helps to ensure that data is always checked and validated before being used in downstream processes.
  • Using data profiling tools: Data profiling tools help to identify patterns, anomalies, and errors in data. They make it easier to identify data quality issues and correct them before they cause downstream problems.

In Python, there are several libraries and tools that can be used for data curation, such as:

  • Great Expectations: A library for automated data validation. It allows data engineers to define expectations about the structure and content of data and validate them automatically.
  • DVC: A version control system for data science and machine learning projects. It allows data scientists to track changes to data files and collaborate with team members.
  • DataProfiler: A library for data profiling and analysis. It allows data engineers to understand the structure and quality of data and identify potential issues.

Other data engineering terms related to
Data Management:


Move rarely accessed data to a low-cost, long-term storage solution to reduce costs. store data for long-term retention and compliance.


Add new data or information to an existing dataset to enhance its value. Enhance data with additional information or attributes to enrich analysis and reporting.


Create a copy of data to protect against loss or corruption.


Identify and remove duplicate records or entries to improve data quality.


Analyzing the number of features or attributes in the data to improve performance.


Enhance data with additional information from external sources.


Extract data from a system for use in another system or application.


Create an optimized data structure for fast search and retrieval.


combine data from different sources to create a unified view for analysis or reporting.


Store the results of expensive function calls and reusing them when the same inputs occur again.


Combine data from multiple datasets into a single dataset.


Extract useful information, patterns or insights from large volumes of data using statistics and machine learning.


Create a conceptual representation of data objects.


Track data processing metrics and system health to ensure high availability and performance.

Named Entity Recognition

Locate and classify named entities in text into pre-defined categories.


Interpret and convert data from one format to another.


Divide data into smaller subsets for improved performance.


Transform your data so it is fit-for-purpose.


Transform raw data before data analysis or machine learning modeling.


Create a copy of data for redundancy or distributed processing.


Increasing the capacity or performance of a system to handle more data or traffic.

Schema Mapping

Translate data from one schema or structure to another to facilitate data integration.


Ensure that data in different systems or databases are in sync and up-to-date.


Check data for completeness, accuracy, and consistency.


Maintain a history of changes to data for auditing and tracking purposes.