Dagster 1.11: dg + Components

Significantly improved pipeline-building experience with Components and dg, enhanced orchestration capabilities, integration power-ups, and more.

Introducing Dagster 1.11 “Build Me Up Buttercup”, shaped hand-in-hand with design partners and the community!

Release Highlights:

Components (stable) brings configurable, reusable building blocks in YAML or Python that declare Dagster definitions without boilerplate.
dg CLI (stable) is new command-line companion scaffolds code, accelerates development, and keeps large projects manageable.
Cross-platform create-dagster command to spin up an opinionated project or workspace layouts in seconds.
Various quality-of-life boosts across core orchestration and the UI.‍
Ecosystem power-ups: Iceberg, dbt Cloud, Airflow, Fivetran, and an in-app Integrations Marketplace.
…and more!

Read on for the details.

Components (stable): Configurable, Reusable Building Blocks for Data Pipelines

Components deliver configurable, reusable building blocks in YAML or Python, letting you define Dagster assets and pipelines effortlessly.

Plug-and-play in YAML or Python: Rapidly build pipelines from concise YAML or lightweight Python that lets you spin up arbitrary Dagster definitions (such as assets, resources, schedules, checks, and more) with almost no boilerplate. Ready-made components ship for dbt, Fivetran, Airbyte, Sling, DLT, Power BI, and more. Check out Components documentation.

Automatic Documentation: Metadata you already write auto-generates live reference pages and inline help, keeping builders unblocked and sparing you the README grind.

Low-code, High-clarity DSL
- Custom Components: If you’re a data platform team serving many data users, wrap any internal script or third-party tool in a custom component and hand it over with the same polished tooling as the built-ins. No extra glue, no re-training your stakeholders.
- Powerful tooling: High-quality errors, strongly typed definitions, and robust tooling integrated into your CI/CD workflows, for robust YAML management.
- Pythonic Templating: Want more customization? Register reusable variables and Python helpers for extensive customization; this allow your stakeholders to craft entire workflows in one defs.yaml, without touching boilerplate.

dg (Stable): Your All-in-One CLI

Think of dg as an IDE you run from your terminal: scaffold code, spin up a local Dagster with UI, launch jobs, and introspect definitions—all behind one command.

Code generation: dg scaffold quickly generates definitions such as assets, components, and more, with zero boilerplate.
Local development & ad-hoc execution: dg dev launches a full Dagster instance + UI in one command. dg launch kicks off jobs or assets right from the CLI for quick ad-hoc execution.
Introspection & validation: View and validate definitions easily with dg list and dg check.
DX utilities: Easily configure VSCode/Cursor extensions, generate schemas, and inspect components.

All commands you're familiar with in the existing dagster CLI are also available here. The dg CLI will eventually supersede the existing dagster CLI entirely, offering enhanced usability and a unified experience. (Note: Certain sub-commands are still forthcoming and will be added soon.)

create-dagster: One-shot project bootstrapper

Instantly set up production-ready Dagster projects or workspaces with one simple command and no dependencies required. This modernized command supersedes the previous scaffold workflow, offering:

A standardized Python directory layout.
Preconfigured local dg CLI setup.
Workspace scaffolding capability (newly supported).
No active Python environment required (pipx, uvx, brew, curl friendly).

UI

Unified asset selection: A flexible, expressive syntax to easily define selections across your assets. Powers alerts, insights, and saved views. Learn more in https://dagster.io/blog/updated-asset-selection-syntax
Runs › Backfills consolidates all backfill activity under the Runs page for faster navigation.
Enhanced Asset Graph: Redesigned, customizable nodes with detailed health indicators.

Core Orchestration Enhancements

Improved Partial Retries: A new re-execution option enabled for re-running only failed assets within multi-asset steps, rather than all asset assets within a failed step. This saves resources and time in heavy dbt execution, multi-asset factories.
Checks in Ops: You can now use asset checks with Ops. Fits your operational and dynamic workflows that don’t fit in Assets paradiam.
Hooks in Assets: Success/failure callbacks now available in assets.
Efficient Backfills & Concurrency Management: Improved backfill policies and multi-threading support enhance scheduling efficiency; run blocking is now default, preventing oversubscription.

Integrations

Fivetran integration GA: the FivetranWorkspace resource is now GA [docs].
dbt Cloud (Beta): first-class job launches and lineage capture [docs].
Apache Iceberg (Preview): Iceberg IOManager writes/reads lake-house tables [docs].
Airflow (Beta): Airflow Component lets you surface Airflow DAGs inside Dagster for mixed-orchestrator observability [docs].
Integrations Marketplace (Preview): “Integrations” tab to browse first- and third-party integrations natively in Dagster UI (enable via User Settings → “Display integrations marketplace”).

Acknowledgments

Huge thanks to everyone who opened issues, filed PRs, tested previews, and cheered us on. Dagster wouldn’t be what it is without you.

Stay tuned for more updates and enhancements in future releases—and, as always, happy data engineering!

‍

We're always happy to hear your feedback, so please reach out to us! If you have any questions, ask them in the Dagster community Slack (join here!) or start a Github discussion. If you run into any bugs, let us know with a Github issue. And if you're interested in working with us, check out our open roles!

Dagster 1.11: Build Me Up Buttercup

Significantly improved pipeline-building experience with Components and dg, enhanced orchestration capabilities, integration power-ups, and more.

Components (stable): Configurable, Reusable Building Blocks for Data Pipelines

dg (Stable): Your All-in-One CLI

create-dagster: One-shot project bootstrapper

UI

Core Orchestration Enhancements

Integrations

Acknowledgments

Dagster Newsletter

Latest writings

dbt Fusion Support Comes to Dagster

What CoPilot Won’t Teach You About Python (Part 2)

Dagster’s MCP Server