.png)
Featuring YAML-based pipeline definitions, plug-and-play integrations, automatic documentation, and a unified CLI experience.
With Dagster 1.11, we introduced a new foundation for building data pipelines: Components and the dg CLI. After months of preview releases, community feedback, and refinements, we’re excited to announce that Components are now generally available.
This update introduces a new YAML-based DSL for defining pipelines, a library of ready-made Components for popular tools, automatic documentation generation for your custom Components, and a powerful CLI to tie it all together. It’s one of our biggest steps yet in making Dagster easier to use and more powerful at the same time.
Why Components?
Components are our answer to a question we’ve heard again and again:
“How do I build pipelines that are simple to author, easy to reuse, and flexible enough to evolve with my platform?”
Until now, data teams often had to choose between ease of use and long-term maintainability. Components bring these worlds together by:
- Encapsulating complexity into clear, composable building blocks.
- Promoting reuse and sharing across teams and projects.
- Standardizing patterns without sacrificing flexibility.
Configurable, Reusable Building Blocks in YAML or Python
Dagster Components deliver configurable, reusable pipeline building blocks that let you declare assets, resources, schedules, and more with almost no boilerplate. Instead of writing lengthy Python code for every asset or job, you can now define them in a few lines of YAML or lightweight Python. Under the hood, each Component is a template that knows how to produce Dagster definitions from configuration.
(Be sure to check out the Components section of the tutorial!)
For example, setting up a dbt project can be as simple as writing a short YAML snippet. Just specify the project path and how to map dbt models to asset keys, and the Component generates all the assets for you. Want to ingest data from an API or run a SQL transformation? Dagster comes with ready-made Components for common integrations like dbt, Fivetran, Airbyte, Sling, dlt, Power BI, and more. These plug-and-play Components let you spin up new pipelines in minutes by filling out a few config fields, rather than hand-coding new assets from scratch.
1type: dagster_dbt.DbtProjectComponent
2
3attributes:
4 project: '{{ project_root }}/dbt'
5 select: "customers"
And if you prefer to work in Python, you can reference Components directly in Python to compose assets, or even write your pipeline logic as a custom Component class. Whether you author in YAML or Python, your pipelines are easier to read, easier to reuse, and faster to build.
Low-Code Simplicity, High-Code Flexibility
Making Dagster more accessible doesn't mean limiting its power. Components provide a low-code approach without a low ceiling. You can define your pipelines using simple YAML, and when you need to go further, you have the full flexibility of Python at your disposal.
You can build your own Components to encapsulate internal scripts or third-party APIs your organization relies on. Wrap your custom logic in a Component class once, and then anyone on your team can use it via YAML with the same polished experience as a built-in integration. No extra glue code, no re-training your stakeholders on new frameworks—just higher-level building blocks that encapsulate best practices.
Components also include a powerful templating system to bridge the gap between YAML and code. You can register reusable template variables and helper functions (“UDFs”) that become available inside your YAML configs. This means you can inject dynamic values or even call functions within a YAML field to handle advanced logic. In short, you get the convenience of YAML for most use cases, with an easy escape hatch to Python when deeper control is required.
Automatic Documentation and Built‑In Help
Keeping pipeline code and documentation in sync can be a chore. Components turn this on its head by generating documentation for you, automatically. The same metadata you provide in your Component definitions is used to produce always-up-to-date reference docs. In practice, this means every Component and its configuration options are self-documenting. Users can browse available Components, see their purpose and parameters, and even get tooltips or CLI help explaining each field.
Whether you’re a new user figuring out how to configure a Snowflake ingestion, or a platform engineer reviewing a teammate’s YAML, the information you need is always at your fingertips. The Dagster UI and CLI (via dg) both leverage this metadata to surface docs. For instance, you can run dg components list to see available Components and their descriptions, or use the web UI to browse auto-generated docs for each Component type. This tight integration of code and documentation means fewer questions, faster onboarding, and shared understanding across your team.
Incremental Adoption, No Disruption
We designed Components to be opt-in and incrementally adoptable. If you already have a Dagster project, you can introduce Components gradually. For example, start by adding one Component-defined asset to your existing repository, or use dg for a new subproject. There are no breaking changes to the core Dagster APIs; everything you’ve built continues to work as before. Components simply add a new layer that you can leverage at your own pace.
Under the hood, Dagster still treats Component-generated assets and resources just like any other. This means you can mix and match: use Components for some parts of your pipeline and pure Python for others. Early adopters have successfully retrofitted Components into large existing deployments, and have had success in scaling and enabling their data teams. In fact, our own team migrated Dagster’s internal data platform to use Components, and saw first-hand how it simplifies project organization and boosts development speed.
Get Started with Dagster Components
Starting with Dagster 1.11.10, Components and the dg CLI are officially marked as generally available, and considered ready for production use. Be sure to check out the Components documentation, for tutorials on building pipelines with Components, creating custom Components, using template variables, and more.
Dagster Components represent a new chapter in simplifying data orchestration. By combining the ease of configuration with the full power of code, we hope to enable more people in your organization to build and own data pipelines. We can’t wait to see what you build with Components now that they’re GA. As always, we welcome your feedback and stories.
Happy building! 🚀
Additional resources
Check out the following links to learn more, and join the Slack and GitHub communities if you have any qustions or feedback!
