In 0.10.0, we introduce dynamic orchestration, a new backfill UI, and support for tracking asset lineage.

Dagster lets a small data infrastructure team efficiently ship a data platform that supports users with different skillsets, letting anyone contribute with minimal coordination required.

In 0.10.0, we introduce unique event-based scheduling capabilities, hardened deployments on Kubernetes, and new primitives for persistence.

Running pipelines is only part of the operational burden of running a data platform. We also need to manage the platform itself and control associated technical debt. We found that Dagster was a very natural place to do that work, with the advantage that our entire operational view of the platform is consolidated in a single tool.

What we’re aiming for with Dagster is a completely horizontal view of our data assets. Our analysts will be able to look up when a raw data ingest from Stitch occurred, when a dbt model ran, or when a plot was generated by a Jupyter notebook and posted in Slack, through a single portal — a single "pane of glass."

People sometimes ask us — should I use Dagster, or should I use dbt? We view Dagster and dbt as complementary technologies, not competing technologies.

Dagster’s support for custom data types helped us achieve better correctness and reliability in our data ingest process, which meant less downstream breakage and better recovery when bad data made it through.

Adopting Dagster was a catalyst for the transformation of our data platform team. We hope our experience is encouraging to other teams facing similar challenges and opportunities.

Spark has a beautiful API — so why is developing with it such a pain?

Our monthly community meeting featured a retrospective of our 0.9.0 release, a preview of our 0.10.0 roadmap, and Prezi's journey from a homegrown orchestration solution to Dagster.

We’re thrilled to announce a new integration between Dagster and a fellow open-source project, Great Expectations (GE).

Dagster is a new type of workflow engine: a data orchestrator. Moving beyond just managing the ordering and physical execution of data computations, Dagster introduces a new primitive: a data-aware, typed, self-describing, logical orchestration graph.

With 0.7.0 we set out improve the Dagster experience with large, production-scale pipelines, deployable to Kubernetes.

With 0.6.0, Dagster comes “batteries-included” — but still with pluggable options — for everything you need to execute, monitor, schedule, deploy, and debug your data applications.

Today the team at Elementl is proud to announce an early release of Dagster, an open-source library for building systems like ETL processes and ML pipelines. We believe they are, in reality, a single class of software system. We call them data applications.