The Dagster platform
Dagster is a next-generation open source orchestration platform for the development, production, and observation of data assets.
Dagster as a productivity platform
Software-Defined Assets (SDAs) are a core concept in the Dagster framework. Working with SDAs is not mandatory, but they add a whole new dimension to the orchestration layer. SDAs allow you to:
- • Manage complexity in your data environment.
- • Write reusable, low-maintenance code.
- • Gain greater control and insights across your pipelines and projects.
Make your team more productive and your code easier to maintain
Build a maintainable and performant data platform
Dagster as a robust orchestration engine
Put your pipelines into production with a robust multi-tenant, multi-tool engine that scales technically and organizationally.
Dagster’s server model eliminates common points of failure
Isolated codebases and environmental requirements, and new processes on each run make Dagster more robust and less prone to failures.
Real-time/event-based, fault-tolerant run scheduling
Dagster’s flexible architecture (K8s, ECS, Docker, etc.) opens up your infrastructure options.
Dagster as a unified control plane
The ‘single plane of glass’ data teams love to use
Rein in the chaos and maintain control over your data as the complexity scales. Centralize your metadata in one tool with built-in observability, diagnostics, cataloging, and lineage. Spot any issues and identify performance improvement opportunities.
Run Timeline provides a full instance overview with real-time status in one elegant UI.
Easily configure and launch runs with Dagster’s Launchpad and its powerful editor. Start runs as defined or easily experiment with their configurations and observe the result.
Automatically maintain an audit of all assets with the built-in Asset Catalog. Once built and defined, any stakeholder can start, inspect, and manage their own runs.
Detect issues and act with surgical precision
Get details on each asset: Freshness, status, schema, metadata, and dependencies displayed in one consolidated view. Model and organize thousands of assets, giving you plenty of room to grow.
Partitions and Backfills
Manage complexity with Partitions and Backfills. Audit historical runs at a granular level.
Observe and assess individual runs and the state of assets generated. Query logs for detailed diagnostics, discover the most time-consuming tasks via a Gantt chart, re-execute subsets of a run, and more.
Automate and integrate your work
Automate your efforts with scriptable alerting, asset-level status reports, and Dagster’s fully extensible alerting framework.
Integrate your infrastructure and monitoring tools with the most versatile and extensible platform.