Introducing Dagster 1.11 “Build Me Up Buttercup”, shaped hand-in-hand with design partners and the community!
Release Highlights:
- Components (stable) brings configurable, reusable building blocks in YAML or Python that declare Dagster definitions without boilerplate.
- dg CLI (stable) is new command-line companion scaffolds code, accelerates development, and keeps large projects manageable.
- Cross-platform create-dagster command to spin up an opinionated project or workspace layouts in seconds.
- Various quality-of-life boosts across core orchestration and the UI.
- Ecosystem power-ups: Iceberg, dbt Cloud, Airflow, Fivetran, and an in-app Integrations Marketplace.
- …and more!
Read on for the details.
Components (stable): Configurable, Reusable Building Blocks for Data Pipelines
Components deliver configurable, reusable building blocks in YAML or Python, letting you define Dagster assets and pipelines effortlessly.
- Plug-and-play in YAML or Python: Rapidly build pipelines from concise YAML or lightweight Python that lets you spin up arbitrary Dagster definitions (such as assets, resources, schedules, checks, and more) with almost no boilerplate. Ready-made components ship for dbt, Fivetran, Airbyte, Sling, DLT, Power BI, and more. Check out Components documentation.
- Automatic Documentation: Metadata you already write auto-generates live reference pages and inline help, keeping builders unblocked and sparing you the README grind.
- Low-code, High-clarity DSL
- Custom Components: If you’re a data platform team serving many data users, wrap any internal script or third-party tool in a custom component and hand it over with the same polished tooling as the built-ins. No extra glue, no re-training your stakeholders.
- Powerful tooling: High-quality errors, strongly typed definitions, and robust tooling integrated into your CI/CD workflows, for robust YAML management.
- Pythonic Templating: Want more customization? Register reusable variables and Python helpers for extensive customization; this allow your stakeholders to craft entire workflows in one defs.yaml, without touching boilerplate.
dg (Stable): Your All-in-One CLI
Think of dg as an IDE you run from your terminal: scaffold code, spin up a local Dagster with UI, launch jobs, and introspect definitions—all behind one command.
- Code generation: dg
scaffoldquickly generates definitions such as assets, components, and more, with zero boilerplate. - Local development & ad-hoc execution: dg
launchkicks off jobs or assets right from the CLI for quick ad-hoc execution. - Introspection & validation: View and validate definitions easily with dg list and dg check.
- DX utilities: Easily configure VSCode/Cursor extensions, generate schemas, and inspect components.
All commands you're familiar with in the existing dagster CLI are also available here. The dg CLI will eventually supersede the existing dagster CLI entirely, offering enhanced usability and a unified experience. (Note: Certain sub-commands are still forthcoming and will be added soon.)
create-dagster: One-shot project bootstrapper
Instantly set up production-ready Dagster projects or workspaces with one simple command and no dependencies required. This modernized command supersedes the previous scaffold workflow, offering:
- A standardized Python directory layout.
- Preconfigured local dg CLI setup.
- Workspace scaffolding capability (newly supported).
- No active Python environment required (pipx, uvx, brew, curl friendly).
UI
- Unified asset selection: A flexible, expressive syntax to easily define selections across your assets. Powers alerts, insights, and saved views. Learn more in https://dagster.io/blog/updated-asset-selection-syntax
- Runs › Backfills consolidates all backfill activity under the Runs page for faster navigation.
- Enhanced Asset Graph: Redesigned, customizable nodes with detailed health indicators.
Core Orchestration Enhancements
- Improved Partial Retries: A new re-execution option enabled for re-running only failed assets within multi-asset steps, rather than all asset assets within a failed step. This saves resources and time in heavy dbt execution, multi-asset factories.
- Checks in Ops: You can now use asset checks with Ops. Fits your operational and dynamic workflows that don’t fit in Assets paradiam.
- Hooks in Assets: Success/failure callbacks now available in assets.
- Efficient Backfills & Concurrency Management: Improved backfill policies and multi-threading support enhance scheduling efficiency; run blocking is now default, preventing oversubscription.
Integrations
- Fivetran integration GA: the
FivetranWorkspaceresource is now GA [docs]. - dbt Cloud (Beta): first-class job launches and lineage capture [docs].
- Apache Iceberg (Preview): Iceberg IOManager writes/reads lake-house tables [docs].
- Airflow (Beta): Airflow Component lets you surface Airflow DAGs inside Dagster for mixed-orchestrator observability [docs].
- Integrations Marketplace (Preview): “Integrations” tab to browse first- and third-party integrations natively in Dagster UI (enable via User Settings → “Display integrations marketplace”).
Acknowledgments
Huge thanks to everyone who opened issues, filed PRs, tested previews, and cheered us on. Dagster wouldn’t be what it is without you.
- Anush (Anush008)
- Alex Noonan (C00ldudeNoonan)
- Dino Scheidt (D1no)
- Exlll (Exlll)
- Fredrik Bakken (FredrikBakken)
- HynekBlaha (HynekBlaha)
- OwenKephart (OwenKephart)
- Pedram Navid (PedramNavid)
- Morgan PERRE (Thenkei)
- Julien "uj" Abadji (Uinelj)
- Andreas Buchmüller (abuchmueller)
- Adam Bloom (adam-bloom)
- Alexander Grueneberg (agrueneberg)
- Alex Langenfeld (alangenfeld)
- Alexandre Potvin-Demers (alexpotv)
- Anurag Anjaria (anuthebananu)
- axelwas (axelwas)
- Stefano (babaMar)
- bartcode (bartcode)
- Ben Gotow (bengotow)
- Ben Pankow (benpankow)
- Josh Braun (braunjj)
- Brian Dailey (briandailey)
- Bruno Bento (brunobbaraujo)
- Charles Augello (chazmo03)
- Claire Lin (clairelin135)
- colton (cmpadden)
- Daniel Gafni (danielgafni)
- David Baker (davvidbaker)
- Deepyaman Datta (deepyaman)
- Dennis Hume (dehume)
- dependabot[bot] (dependabot[bot])
- Christopher DeCarolis (dpeng817)
- Dragos Pop (dragos-pop)
- Edson Nogueira (edsoncezar16)
- Andrew Grigorev (ei-grad)
- Eli Boyarski (eli-b)
- evbevz (evbevz)
- Sergei Nikiforov (futurewasfree)
- Gareth Brickman (garethbrickman)
- Gecko Security (geckosecurity)
- geoHeil (geoHeil)
- gibsondan (gibsondan)
- Guillaume Tauzin (gtauzin)
- Isaac Hellendag (hellendag)
- Isaac Jordan (isaac-jordan)
- jamiedemaria (jamiedemaria)
- jj (jjyeo)
- Joseph McCartin (jmccartin)
- Jordan Sanders (jmsanders)
- Johnny Santamaria (johnny603)
- jonathanunderwood (jonathanunderwood)
- Kate Farrar (katefarrar)
- Ken Geis (kgeis)
- Matt Krukowski (krukowski)
- Bryan Lee (liby)
- Marijn Valk (marijncv)
- Matt Giles (mattgiles)
- Maxime Armstrong (maximearmstrong)
- mfiliba (mfiliba)
- natpatchara-w (natpatchara-w)
- Neil Fulwiler (neilfulwiler)
- Nicolas Estrada (neuromantik33)
- Nikki Everett (neverett)
- Martin Mauch (nightscape)
- Nishan (nishan-soni)
- Pete Hunt (petehunt)
- Tom (petrusek)
- prha (prha)
- raayu83 (raayu83)
- René Petersen (renepetersen1984)
- Ross Housewright (ross-whatnot)
- Marco polo (salazarm)
- Sean Quinlan (sbquinlan)
- Nick Schrock (schrockn)
- seyf97 (seyf97)
- shimon pats (shimon-cherrypick)
- Sean Mackesey (smackesey)
- Paul Stepler-Camp (steplercamp)
- Steven Ayers (stevenayers)
- Mohan Talla (tallamohan)
- Dimitry Foures (thedimlebowski)
- Martin Erpicum (tintamarre)
- Tomek Zbrozek (tomekzbrozek)
- Vinodh Zamboulingame (vzamboulingame)
- yassun7010 (ya7010)
- Yuhan Luo (yuhan)
- Shane Zarechian (zero-stroke)
Stay tuned for more updates and enhancements in future releases—and, as always, happy data engineering!
We're always happy to hear your feedback, so please reach out to us! If you have any questions, ask them in the Dagster community Slack (join here!) or start a Github discussion. If you run into any bugs, let us know with a Github issue. And if you're interested in working with us, check out our open roles!
