Sep 20, 2022

Dagster vs. Airflow

We often get asked why a data team should choose Dagster over Apache Airflow. We compare Dagster and Airflow for data orchestration, in five parts.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Nick Schrock
Name
Nick Schrock
Handle
@schrockn

Dagster Newsletter: Get updates delivered to your inbox

Nov 30, 2022

Getting Stuff Done: a Guide to Productive Software Engineering

To be a more productive software engineer you need to master changes, how these affect the program and others on the ...
Alex Langenfeld
Name
Alex Langenfeld
Handle
@alex_langenfeld
Nov 21, 2022

Safe and Easy: Managing Secrets in Dagster Cloud

Dagster Cloud’s new Environment Variables UI makes it easy to set up scoped environment variables.
Erin Cochran
Name
Erin Cochran
Handle
Daniel Gibson
Name
Daniel Gibson
Handle
Nov 18, 2022

My Path to Elementl - Part 2

Pete Hunt takes over as CEO as Nick Schrock takes on the CTO role.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Nov 11, 2022

Pushing REST-API data to Google Sheets with Dagster

A total beginners tutorial in which we store REST API data in Google Sheets and learn some key abstractions.
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Nov 7, 2022

Adding types to a large Python codebase

We decided to drive Dagster to a 100%-typed public interface. This turned out to be a significant undertaking. Lessons ...
Sean Mackesey
Name
Sean Mackesey
Handle
Nov 2, 2022

Running data science notebooks with Dagster: a Noteable integration

The Noteable team adds major powerups for data scientists looking to orchestrate Notebooks with Dagster
Jamie DeMaria
Name
Jamie DeMaria
Handle
Oct 31, 2022

Orchestrating Machine Learning Pipelines with Dagster

To boost your ML efforts, improve your pipeline as well as your model.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Oct 27, 2022

Orchestrating Data Science at Zephyr AI

Zephyr AI applies data science to massive datasets of DNA and healthcare records to deliver novel AI-driven insights.
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Oct 25, 2022

Build a poor man’s data lake from scratch with DuckDB

DuckDB is so hot right now. Could it replace our cloud data warehouses or data lakes?
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Oct 19, 2022

The Unreasonable Effectiveness of Data Pipeline Smoke Tests

Data practitioners waste time writing unit tests to catch bugs they could have caught with smoke tests.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Oct 17, 2022

Web workers are not the answer

A tale of overstretched logs, counterintuitive web worker behavior, and ultimately a troublesome cursor issue.
Marco Salazar
Name
Marco Salazar
Handle
@BkOptimism
Alex Langenfeld
Name
Alex Langenfeld
Handle
@alex_langenfeld
Oct 16, 2022

Dagster at all 5 steps of the development lifecycle

Dagster facilitates a data engineers work across all five steps in the development lifecycle.
Oct 6, 2022

A Dagster Crash Course

If you are looking to get up and running with Dagster in 10 minutes or less, this is a good place to start. Buckle up.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Oct 4, 2022

Postgres: a better message queue than Kafka?

When lots of event logs must be stored and indexed, Kafka is the obvious choice. Naturally, our queue runs on Postgres.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Aug 24, 2022

How EvolutionIQ rebuilt its ML platform for enormous productivity.

A guide for CIOs/CTOs and engineering leaders looking to master the Modern Data Stack and develop a high performance ...
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Aug 18, 2022

Our Pricing Philosophy for Dagster Cloud

Our straightforward usage-based pricing aims to encourage practitioners to work with the framework, not against it.
Rex Ledesma
Name
Rex Ledesma
Handle
@_rexledesma
Aug 17, 2022

Spend less time debugging with Dagster

It’s not uncommon for a data engineer to devote 80% of their day to debugging. Dagster radically improves on this.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Owen Kephart
Name
Owen Kephart
Handle
Aug 9, 2022

Launching Dagster Cloud to GA

The enterprise orchestration platform that puts developer experience first: hybrid or serverless deployments, native ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Aug 5, 2022

Introducing Dagster 1.0: Hello

Announcing Dagster 1.0. - a stable foundation for building the orchestration layer for modern data platforms.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Aug 3, 2022

The Open Core Business Model

The relationship between Dagster, the open-source project, and Dagster Cloud, our hosted SaaS platform.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Jul 26, 2022

Dagster Cloud goes SOC 2

Elementl, the company behind the Dagster data orchestration tool achieves SOC2 compliance.
Selina Li
Name
Selina Li
Handle
Jul 25, 2022

Dagster Day: Announcing Dagster 1.0 and Dagster Cloud

The release of Dagster 1.0 and the GA launch of Dagster Cloud represent major milestones in the evolution of our ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Jul 12, 2022

Roman roads in data engineering: don't write data pipelines from scratch

Work in a way that lays the foundation for your next data product while you're building your current one.
Claire Lin
Name
Claire Lin
Handle
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Jun 23, 2022

The Data Exchange: software-defined assets

Nick Schrock on software-defined assets, a new approach to managing, maintaining, and orchestrating data declaratively.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Jun 22, 2022

My Path to Elementl: Pete Hunt

Pete Hunt discusses what caused him to make the leap from Twitter to Elementl.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Jun 20, 2022

Orchestrating Python and dbt with Dagster

How asset-focused orchestration bridges the gap between some of data's most popular tools.
Owen Kephart
Name
Owen Kephart
Handle
Jun 15, 2022

Dagster 0.15.0: Cool for the Summer

In 0.15.0, software-defined assets are now marked fully stable and are ready for primetime.
Mollie Pettit
Name
Mollie Pettit
Handle
@MollzMP
Mar 9, 2022

New in 0.14.0: Dagster-Airbyte Integration

0.14.0 introduces a deep integration with Airbyte: view Airbyte logs directly in Dagit, and every updated table will be ...
Owen Kephart
Name
Owen Kephart
Handle
Mar 1, 2022

Introducing Software-Defined Assets

Software-Defined Assets are a transformative new abstraction that allows data teams to focus on the end-product not the ...
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Mar 1, 2022

Dagster 0.14.0: Table Schema API + Pandera Integration

Introducing two asset observability-enhancing features: Table Schema API, and an integration with the dataframe ...
Sean Mackesey
Name
Sean Mackesey
Handle
Mar 1, 2022

Dagster 0.14.0: Never Felt Like This Before

We’re thrilled to release version 0.14.0 of Dagster. This version introduces much more mature version of ...
Mollie Pettit
Name
Mollie Pettit
Handle
@MollzMP
Feb 17, 2022

Rebundling the Data Platform

'The Unbundling of Airflow' argued that modern data stack solutions (data ingestion, data transformation, reverse ETL) ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Dec 2, 2021

Introducing Dagster Cloud

Dagster Cloud, the enterprise orchestration platform that puts developer experience first, with fully serverless or ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Nov 20, 2021

Laying the foundation of your data platform for the era of big complexity

Listen to founder and CEO Nick Schrock talk about how Dagster helps tame the complexity and scale when working with ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Nov 17, 2021

Hello Big Complexity: Is Your Modern Data Stack Ready?

Listen to Nick Schrock discuss the evolution of data from Big Data to Big Complexity in this episode of the Mad Data ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Nov 16, 2021

Why Elementl and Dagster: The Decade of Data

Announcing our $14M Series A led by Index Ventures, alongside Sequoia Capital, Slow Ventures, Coatue, Amplify Partners, ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Nov 8, 2021

New in Dagster 0.13.0: Logging Improvements!

Logging without context, instance-wide handlers, capturing python logs, and more! Learn about the improvements we've ...
Owen Kephart
Name
Owen Kephart
Handle
Oct 28, 2021

Dagster 0.13.0: A New Foundation

We’re proud to announce 0.13.0 of Dagster with dramatic improvements to our core APIs, completely revamped UI, and ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Aug 10, 2021

Community Memo: Moving Dagster's Core APIs Towards 1.0

Dagster commits to a stable set of production-ready APIs for building solid data platforms.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Jul 19, 2021

Dagster 0.12.0: Into the Groove

In 0.12.0, we introduce pipeline failure sensors, solid-level retries, and more convenient testing APIs.
Owen Kephart
Name
Owen Kephart
Handle
May 25, 2021

Community Memo: Approachability Improvements

In the last two months, we've made a set of changes aimed at making Dagster more approachable: to smooth out its ...
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
May 18, 2021

Incrementally Adopting Dagster at Mapbox

At Mapbox, we've adopted Dagster without breaking compatibility with our legacy Airflow systems -- and with huge gains ...
Ben Pleasanton
Name
Ben Pleasanton
Handle
May 13, 2021

Moving past Airflow: Why Dagster is the next-generation data orchestrator

A comparison between Dagster and Airflow. Here we detail the differences between the two systems, and make the case for ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Apr 1, 2021

Dagster 0.11.0: Lucky Star

In 0.11.0, we introduce dynamic orchestration, a new backfill UI, and support for tracking asset lineage.
Mar 15, 2021

Building shared spaces for data teams at Drizly

Our small data infrastructure team built a data platform that supports users with different skillsets, letting anyone ...
Dennis Hume
Name
Dennis Hume
Handle
Jan 19, 2021

Dagster 0.10.0: The Edge of Glory

In 0.10.0, we introduce unique event-based scheduling capabilities, hardened deployments on Kubernetes, and new ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Max Gasner
Name
Max Gasner
Handle
@gasnerpants
Dec 9, 2020

Good Data at Good Eggs: Using Dagster to manage the data platform

Running pipelines is only part of running a data platform. We need to manage the platform and control technical debt. ...
David Wallace
Name
David Wallace
Handle
@davidjwallace
Nov 5, 2020

Good Data at Good Eggs: Data observability with the asset catalog

Dagster gives us a single "pane of glass" for data assets. Analysts can look up when a Stitch raw data ingest occurred, ...
David Wallace
Name
David Wallace
Handle
@davidjwallace
Oct 29, 2020

Dagster and dbt: Better Together

People sometimes ask us — should I use Dagster, or should I use dbt? We view Dagster and dbt as complementary ...
AJ Nadel
Name
AJ Nadel
Handle
@AJ_Nadel
Bob Chen
Name
Bob Chen
Handle
@bobchen168
Oct 1, 2020

Good Data at Good Eggs: Data infrastructure correctness and reliability

Dagster’s custom data types helped achieve correctness and reliability in our data ingest process, less downstream ...
David Wallace
Name
David Wallace
Handle
@davidjwallace
Oct 1, 2020

Good Data at Good Eggs: Part 1 of 4

Adopting Dagster transformed our data platform team. We hope our experience is encouraging to other teams facing ...
David Wallace
Name
David Wallace
Handle
@davidjwallace
Sep 16, 2020

Testing and Deploying PySpark Jobs with Dagster

Spark has a beautiful API but developing with it is a pain because different stages of development and deployment ...
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Sep 15, 2020

Community Memo: September 2020 Update

A retrospective of our 0.9.0 release, a preview of our 0.10.0 roadmap, and Prezi's journey from a homegrown ...
Sep 10, 2020

Great Expectations for Dagster

We’re thrilled to announce a new integration between Dagster and a fellow open-source project, Great Expectations (GE).
Leor Fishman
Name
Leor Fishman
Handle
@fishmanl
Aug 25, 2020

Forward Thinking Leaders

Nick Schrock shares insights on how to on how to sell new tech concepts to developers.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Aug 11, 2020

Dagster: The Data Orchestrator

As a workflow engine, Dagster moves beyond ordering and executing data computations. It introduces a new primitive: a ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Max Gasner
Name
Max Gasner
Handle
@gasnerpants
Feb 26, 2020

Dagster 0.7.0: Waiting To Exhale

With 0.7.0 we set out improve the Dagster experience with large, production-scale pipelines, deployable to Kubernetes.
Oct 10, 2019

Dagster 0.6.0: Impossible Princess

Dagster 0.6.0 comes “batteries-included” and pluggable options to execute, monitor, schedule, deploy, and debug your ...
Jul 8, 2019

Introducing Dagster

Elementl announces an early release of Dagster, an open-source library for building ETL processes, ML pipelines and ...