Blog | Dagster

23 Posts

Filter Posts

Why Elementl and Dagster: The Decade of Data

Announcing our $14M Series A led by Mike Volpi of Index Ventures, alongside Sequoia Capital, Slow Ventures, Coatue, Amplify Partners, OSS Capital, and other leading investors.

Nick Schrock

Name: Nick Schrock
Handle

New in Dagster 0.13.0: Logging Improvements!

Logging without context, instance-wide handlers, capturing python logs, and more! Learn about the improvements we've made to Dagster logging since 0.12.0.

Owen Kephart

Name: Owen Kephart
Handle

Dagster 0.13.0: A New Foundation

We’re proud to announce 0.13.0 of Dagster. We’ve made dramatic improvements to our core APIs, completely revamped our UI, and brought renewed clarity to our mission.

Nick Schrock

Name: Nick Schrock
Handle

Community Memo: Moving Dagster's Core APIs Towards 1.0

As data practitioners increasingly depend on Dagster as a critical piece of their platforms, we believe it’s important to commit to a stable set of APIs that they can expect to remain the same for a very long time.

Sandy Ryza

Name: Sandy Ryza
Handle

Dagster 0.12.0: Into the Groove

In 0.12.0, we introduce pipeline failure sensors, solid-level retries, and more convenient testing APIs.

Owen Kephart

Name: Owen Kephart
Handle

Community Memo: Approachability Improvements

In the last two months, we've made a set of changes aimed at making Dagster more approachable: to smooth out its learning curve and reduce its boilerplate.

Sandy Ryza

Name: Sandy Ryza
Handle

Incrementally Adopting Dagster at Mapbox

At Mapbox, we've adopted Dagster without breaking compatibility with our legacy Airflow systems -- and with huge gains to developer productivity.

Ben Pleasanton

Name: Ben Pleasanton
Handle

Moving past Airflow: Why Dagster is the next-generation data orchestrator

Our most requested piece of content is a comparison between Dagster and Airflow. Here we detail the differences between the two systems, and make the case for choosing Dagster.

Nick Schrock

Name: Nick Schrock
Handle

Dagster 0.11.0: Lucky Star

In 0.11.0, we introduce dynamic orchestration, a new backfill UI, and support for tracking asset lineage.

Building shared spaces for data teams at Drizly

Dagster lets a small data infrastructure team efficiently ship a data platform that supports users with different skillsets, letting anyone contribute with minimal coordination required.

Dennis Hume

Name: Dennis Hume
Handle

Dagster 0.10.0: The Edge of Glory

In 0.10.0, we introduce unique event-based scheduling capabilities, hardened deployments on Kubernetes, and new primitives for persistence.

Nick Schrock

Name: Nick Schrock
Handle

Max Gasner

Name: Max Gasner
Handle: @gasnerpants

Good Data at Good Eggs: Using Dagster to manage the data platform

Running pipelines is only part of the operational burden of running a data platform. We also need to manage the platform itself and control associated technical debt. We found that Dagster was a very natural place to do that work, with the advantage that our entire operational view of the platform is consolidated in a single tool.

David Wallace

Name: David Wallace
Handle

Good Data at Good Eggs: Data observability with the asset catalog

What we’re aiming for with Dagster is a completely horizontal view of our data assets. Our analysts will be able to look up when a raw data ingest from Stitch occurred, when a dbt model ran, or when a plot was generated by a Jupyter notebook and posted in Slack, through a single portal — a single "pane of glass."

David Wallace

Name: David Wallace
Handle

Dagster and dbt: Better Together

People sometimes ask us — should I use Dagster, or should I use dbt? We view Dagster and dbt as complementary technologies, not competing technologies.

AJ Nadel

Name: AJ Nadel
Handle

Bob Chen

Name: Bob Chen
Handle: @bobchen168

Good Data at Good Eggs: Correctness and reliability for data infrastructure

Dagster’s support for custom data types helped us achieve better correctness and reliability in our data ingest process, which meant less downstream breakage and better recovery when bad data made it through.

David Wallace

Name: David Wallace
Handle

Good Data at Good Eggs: Part 1 of 4

Adopting Dagster was a catalyst for the transformation of our data platform team. We hope our experience is encouraging to other teams facing similar challenges and opportunities.

David Wallace

Name: David Wallace
Handle

Testing and Deploying PySpark Jobs with Dagster

Spark has a beautiful API — so why is developing with it such a pain?

Sandy Ryza

Name: Sandy Ryza
Handle

Dagster Community Update: September 2020

Our monthly community meeting featured a retrospective of our 0.9.0 release, a preview of our 0.10.0 roadmap, and Prezi's journey from a homegrown orchestration solution to Dagster.

Great Expectations for Dagster

We’re thrilled to announce a new integration between Dagster and a fellow open-source project, Great Expectations (GE).

Leor Fishman

Name: Leor Fishman
Handle

Dagster: The Data Orchestrator

Dagster is a new type of workflow engine: a data orchestrator. Moving beyond just managing the ordering and physical execution of data computations, Dagster introduces a new primitive: a data-aware, typed, self-describing, logical orchestration graph.

Nick Schrock

Name: Nick Schrock
Handle

Max Gasner

Name: Max Gasner
Handle: @gasnerpants

Dagster 0.7.0: Waiting To Exhale

With 0.7.0 we set out improve the Dagster experience with large, production-scale pipelines, deployable to Kubernetes.

Dagster 0.6.0: Impossible Princess

With 0.6.0, Dagster comes “batteries-included” — but still with pluggable options — for everything you need to execute, monitor, schedule, deploy, and debug your data applications.

Introducing Dagster

Today the team at Elementl is proud to announce an early release of Dagster, an open-source library for building systems like ETL processes and ML pipelines. We believe they are, in reality, a single class of software system. We call them data applications.