Listen to Nick Schrock talk about how Dagster helps tame the complexity and scale when working with data in this episode of the Data Engineering podcast

Listen to Nick Schrock discuss the evolution of data from Big Data to Big Complexity in this episode of the Mad Data podcast

As data practitioners increasingly depend on Dagster as a critical piece of their platforms, we believe it’s important to commit to a stable set of APIs that they can expect to remain the same for a very long time.

In the last two months, we've made a set of changes aimed at making Dagster more approachable: to smooth out its learning curve and reduce its boilerplate.

Our most requested piece of content is a comparison between Dagster and Airflow. Here we detail the differences between the two systems, and make the case for choosing Dagster.

In 0.10.0, we introduce unique event-based scheduling capabilities, hardened deployments on Kubernetes, and new primitives for persistence.

Running pipelines is only part of the operational burden of running a data platform. We also need to manage the platform itself and control associated technical debt. We found that Dagster was a very natural place to do that work, with the advantage that our entire operational view of the platform is consolidated in a single tool.

What we’re aiming for with Dagster is a completely horizontal view of our data assets. Our analysts will be able to look up when a raw data ingest from Stitch occurred, when a dbt model ran, or when a plot was generated by a Jupyter notebook and posted in Slack, through a single portal — a single "pane of glass."

People sometimes ask us — should I use Dagster, or should I use dbt? We view Dagster and dbt as complementary technologies, not competing technologies.

Dagster’s support for custom data types helped us achieve better correctness and reliability in our data ingest process, which meant less downstream breakage and better recovery when bad data made it through.

Adopting Dagster was a catalyst for the transformation of our data platform team. We hope our experience is encouraging to other teams facing similar challenges and opportunities.

Spark has a beautiful API — so why is developing with it such a pain?

We’re thrilled to announce a new integration between Dagster and a fellow open-source project, Great Expectations (GE).

Dagster is a new type of workflow engine: a data orchestrator. Moving beyond just managing the ordering and physical execution of data computations, Dagster introduces a new primitive: a data-aware, typed, self-describing, logical orchestration graph.