Dagster Data Engineering Glossary:
Data Engineering Terms Explained
Terms and Definitions You Need to Know as a Data Engineer
Back to Glossary IndexACID Properties
The set of properties of database transactions intended to guarantee validity even in the event of errors or failures, encompassing Atomicity, Consistency, Isolation, and Durability.
See WikipediaAgile Methodology
An iterative approach to software development and project management that prioritizes flexibility and customer satisfaction, often used by data engineering teams to manage projects.
See WikipediaAlation
A machine learning data catalog that helps people find, understand, and trust the data.
Vendor WebsiteAligning
Aligning data can mean one of three things: aligning datasets, meeting business rules, or arranging data elements in memory.
See Glossary entryApache Atlas
A scalable and extensible set of core foundational governance services, enabling enterprises to effectively and efficiently meet their compliance requirements within Hadoop.
Project websiteArgo
An open-source container-native workflow engine for orchestrating parallel jobs on Kubernetes.
Vendor WebsiteAssociation Rule Mining
A machine learning method aimed at identifying interesting relations between variables (items or events) in large databases, frequently used for market basket analysis.
Find out moreBackend-as-a-Service (BaaS)
A cloud computing service model that serves as the middleware that provides developers with ways to connect their web and mobile applications to cloud services via application programming interfaces (APIs) and software developers' kits (SDKs).
See WikipediaCBOR (Concise Binary Object Representation)
A binary format encoding data in a more efficient and compact manner than JSON. It is designed to efficiently serialize and deserialize complex data structures without losing schema-free property of JSON.
Learn MoreCombinatorial Explosion
A phenomenon in computer science where the number of possible solutions or combinations in a problem grows exponentially with the size of the problem.
Learn MoreCrash Recovery
The process by which an operating system or application restarts operation after a crash, possibly recovering lost data.
Learn MoreCRON
A time-based job scheduler in Unix-like computer operating systems for scheduling periodic jobs at fixed times, dates, or intervals.
Learn MoreCursor
A database object used to traverse the results of a SQL query, allowing individual rows to be accessed.
Learn MoreDAG (Directed Acyclic Graph)
A finite directed graph with no directed cycles, used extensively in representing data flow in data processing systems like Apache Airflow.
Learn MoreData Degradation
The gradual loss or deterioration of data quality over time.
Learn MoreData Dictionary
A collection of descriptions of the data objects or items in a data model for the benefit of programmers and others who need to refer to them.
Learn MoreData Provenance
Information that helps to trace the origins, processing, and use of data, helping to determine the quality and reliability of the dataset.
Learn MoreDrift Detection
Identifying when the statistical properties of the target variable, which the model is trying to predict, change.
Learn MoreElasticsearch
A search engine based on the Lucene library, providing a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.
Learn MoreFeature Scaling
A method used to normalize the range of independent variables or features of data.
Learn MoreFeature Store
A centralized repository for storing, serving, and sharing machine learning features, allowing for the consistent use of features across different models.
Learn More