Spark has a beautiful API — so why is developing with it such a pain?

Our monthly community meeting featured a retrospective of our 0.9.0 release, a preview of our 0.10.0 roadmap, and Prezi's journey from a homegrown orchestration solution to Dagster.

We’re thrilled to announce a new integration between Dagster and a fellow open-source project, Great Expectations (GE).

Dagster is a new type of workflow engine: a data orchestrator. Moving beyond just managing the ordering and physical execution of data computations, Dagster introduces a new primitive: a data-aware, typed, self-describing, logical orchestration graph.

With 0.7.0 we set out improve the Dagster experience with large, production-scale pipelines, deployable to Kubernetes.

With 0.6.0, Dagster comes “batteries-included” — but still with pluggable options — for everything you need to execute, monitor, schedule, deploy, and debug your data applications.

Today the team at Elementl is proud to announce an early release of Dagster, an open-source library for building systems like ETL processes and ML pipelines. We believe they are, in reality, a single class of software system. We call them data applications.