Data Engineering Terms Explained
A guide to key terms used in data engineering. Entries with the icon 
include useful code examples in Python. For installation instructions for the packages used in the examples, visit the packages page. For a complete list of Data Engineering terms all data engineers should know, please check out the terms index.
104Terms
Aggregate
Combine data from multiple sources into a single dataset.
Align
Aligning data can mean one of three things: aligning datasets, meeting business rules, or arranging data elements in memory.
Anomaly Detection
Identify data points or events that deviate significantly from expected patterns or behaviors.
Anonymize
Remove personal or identifying information from data.
Append
Adding or attaching new records or data items to the end of an existing dataset, database table, file, or list.
Archive
Move rarely accessed data to a low-cost, long-term storage solution to reduce costs. Store data for long-term retention and compliance.
AsyncIO
Speed up execution with asynchronous I/O.
Auto-materialize
The automatic execution of computations and the persistence of their results.
Backpressure
A mechanism to handle situations where data is produced faster than it can be consumed.
Backup
Create a copy of data to protect against loss or corruption.
Batch Processing
Process large volumes of data all at once in a single operation or batch.
Categorize
Organizing and classifying data into different categories, groups, or segments.
Checkpointing
Saving the state of a process at certain points so that it can be restarted from that point in case of failure.
Clean or Cleanse
Remove invalid or inconsistent data values, such as empty fields or outliers.
Cluster
Group data points based on similarities or patterns to facilitate analysis and modeling.
Compact
Reducing the size of data while preserving its essential information.
Compress
Reduce the size of data to save storage space and improve processing performance.
Consolidate
Combine multiple datasets into one to create a more comprehensive view of the data.
Cosine Similarity
A measure of similarity between two entities used in text analysis, natural language processing, etc.
Curate
Select, organize, and annotate data to make it more useful for analysis and modeling.
Deduplicate
Identify and remove duplicate records or entries to improve data quality.
De-identify
Remove personally identifiable information (PII) from data to protect privacy and comply with regulations.
Denoise
Remove noise or artifacts from data to improve its accuracy and quality.
Denormalize
Optimize data for faster read access by reducing the number of joins needed to retrieve related data.
Derive
Extracting, transforming, and generating new data from existing datasets.
Deserialize
Deserialization is essentially the reverse process of serialization. See: 'Serialize'.
Dimensionality
Analyzing the number of features or attributes in the data to improve performance.
Discretize
Transform continuous data into discrete categories or bins to simplify analysis.
Downsample
Reduce the amount of data for analysis, storage, or processing.
Encapsulate
The bundling of data with the methods that operate on that data.
Encode
Convert categorical variables into numerical representations for ML algorithms.
Enrich
Enhance data with additional information from external sources.
ETL
Extract, transform, and load data between different systems.
Export
Extract data from a system for use in another system or application.
Extrapolate
Predict values outside a known range, based on the trends or patterns identified within the available data.
Fan-Out
A pipeline design in which one operation is broken into - or results in - many parallel downstream tasks.
Feature Extraction
Identify and extract relevant features from raw data for use in analysis or modeling.
Feature Selection
Identify and select the most relevant and informative features for analysis or modeling.
Filter
Extract a subset of data based on specific criteria or conditions.
Fragment
Break data down into smaller chunks for storage and management purposes.
Geospatial Analysis
Analyze data that has geographic or spatial components to identify patterns and relationships.
Graph Theory
A powerful tool to model and understand intricate relationships within our data systems.
Hash
Convert data into a fixed-length code to improve data security and integrity.
Homogenize
Make data uniform, consistent, and comparable.
Idempotent
An operation that produces the same result each time it is performed.
Impute
Fill in missing data values with estimated or imputed values to facilitate analysis.
Index
Create an optimized data structure for fast search and retrieval.
Ingest
The initial collection and import of data from various sources into your processing environment.
Interpolate
Use known data values to estimate unknown data values.
Lineage
Understand how data moves through a pipeline, including its origin, transformations, dependencies, and ultimate consumption.
Linearizability
Ensure that each individual operation on a distributed system appear to occur instantaneously.
Linearize
Transforming the relationship between variables to make datasets approximately linear.
Mask
Obfuscate sensitive data to protect its privacy and security.
Materialize
Executing a computation and persisting the results into storage.
Memoize
Store the results of expensive function calls and reusing them when the same inputs occur again.
Merge
Combine data from multiple datasets into a single dataset.
Model
Create a conceptual representation of data objects.
Multiprocessing
Optimize execution time with multiple parallel processes.
Munge
See 'wrangle'.
Named Entity Recognition
Locate and classify named entities in text into pre-defined categories.
Interested in trying Dagster+ for free?
Enterprise orchestration that puts developer experience first.
Serverless or hybrid deployments, native branching, and out-of-
the-box CI/CD.