Blog | Dagster: Articles on data engineering and data pipeline orchestration
Mar 11, 2024

Integrate OpenAI calls into your pipelines

The new dagster-openai integration lets you tap into the power of LLMs in a cost-efficient way.
Yuhan Luo
Name
Yuhan Luo
Handle
@yuhan
Maxime Armstrong
Name
Maxime Armstrong
Handle
@maxime

Dagster Newsletter: Get updates delivered to your inbox

Mar 10, 2024

Tech Talks Daily: Data, Decisions, and Dagster

Nick Schrock shares his blueprint for engineering excellence on the Tech Talks Daily Podcast.
Mar 6, 2024

Dagster University Presents: Dagster & dbt™

Learn how to combine your dbt™ knowledge with Dagster’s asset-focused approach for an enhanced data platform experience.
Erin Cochran
Name
Erin Cochran
Handle
Mar 2, 2024

How to Make Data a Team Sport

Enabling internal access and collaboration around data in organizations is vital to tackling data complexity.
Colton Padden
Name
Colton Padden
Handle
@colton
TéJaun RiChard
Name
TéJaun RiChard
Handle
@tejaun
Feb 27, 2024

Breaking Packages in Python

An exposé of the nooks and crannies of Python’s modules and packages.
Pedram Navid
Name
Pedram Navid
Handle
@pdrmnvd
Feb 23, 2024

Balancing the Data Scales: Centralization vs. Decentralization

Learn how organizations can harness the strengths of both approaches to optimize their data operations.
TéJaun RiChard
Name
TéJaun RiChard
Handle
@tejaun
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Feb 20, 2024

BenchSci: A Leap Forward with Dagster

Learn about how BenchSci uses Dagster in their journey to expedite drug development.
TéJaun RiChard
Name
TéJaun RiChard
Handle
@tejaun
Feb 17, 2024

A Geek Leader: Interview with Nick Schrock

John Rouda interviewed Nick Schrock, Founder of Dagster Labs, on open-source, ML, and the future of Dagster.
Feb 15, 2024

Addressing Big Complexity Through Strategic Orchestration

For organizations looking to thrive in the era of Big Complexity, it’s time to reassess the role of orchestration in ...
TéJaun RiChard
Name
TéJaun RiChard
Handle
@tejaun
Feb 14, 2024

Scaling Data Pipelines

Nick joins the Open Source Underdogs podcast for a conversation on how Dagster Labs is evolving.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Feb 8, 2024

Standardize Pipelines with Domain-Specific Languages

By implementing DSLs, data teams can open their data platform to many more users without compromising on standards.
Elliot Gunn
Name
Elliot Gunn
Handle
@elliot
Tim Castillo
Name
Tim Castillo
Handle
@tim
Feb 7, 2024

Learning and Sharing in Public

On the culture of learning and sharing in Data Engineering.
Pedram Navid
Name
Pedram Navid
Handle
@pdrmnvd
Feb 6, 2024

Facebook Eng Culture & Modern Data Stack Consolidation

On open source software, data, and understanding Facebook’s high performance culture.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Feb 5, 2024

Thinking in Assets

How to develop data pipelines using Software-defined Assets.
Tim Castillo
Name
Tim Castillo
Handle
@tim
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Jan 29, 2024

What Dagster Believes About Data Platforms

The beliefs that organizations adopt about the way their data platforms should function influence their outcomes. Here ...
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Jan 26, 2024

Data Driven - The Role of AI and LLMs in Data

Pedram Navid joins the Data Driven podcast to discuss the role of AI and LLMs in data.
Pedram Navid
Name
Pedram Navid
Handle
@pdrmnvd
Jan 26, 2024

Data Driven - Cutting Through the Noise of Data Products

Pedram Navid talks about how data teams can strategically enable self-service to speed up business decisions.
Pedram Navid
Name
Pedram Navid
Handle
@pdrmnvd
Jan 12, 2024

Dagster 1.6: Back to Black

Major UI enhancements, Dagster Pipes upgrades and of course, dark mode :-)
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Jan 10, 2024

Retain.ai joins Dagster Labs

We’re excited and humbled to bring the Retain.ai organization into our fold to help build out Dagster’s data ...
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Jan 3, 2024

Machine Learning Pipelines Are Still Data Pipelines

Sandy Ryza, Lead Engineer at Dagster Labs, talks data engineering for machine learning efforts.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Dec 21, 2023

Alter Everything - The Present & Future of Data Engineering

Nick Schrock joins the Alteryx podcast about data science and analytics culture.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Dec 4, 2023

How Dagster Labs runs Dagster: Open-Sourcing our Own Pipelines

A technical deep dive into the patterns and implementations of the Dagster Open Platform using our open-sourced code ...
Tim Castillo
Name
Tim Castillo
Handle
@tim
Nov 29, 2023

Scaling Dagster’s DAG Visualization to Handle Tens of Thousands of Assets

How the Dagster frontend team rapidly scaled Dagster’s DAG visualization for enterprise-sized data asset graphs.
Marco Salazar
Name
Marco Salazar
Handle
@BkOptimism
Nov 28, 2023

Abstracting Pipelines for Analysts with a YAML DSL

How SimpliSafe’s small engineering team uses YAML DSL within Dagster’s powerful data platform to support analysts and ...
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Nov 20, 2023

High-performance Python for Data Engineering

Learn how to optimize your Python data pipeline code to run faster with our high-performance Python guide for data ...
Elliot Gunn
Name
Elliot Gunn
Handle
@elliot
Nov 14, 2023

That Tech Pod

The Journey from Engineer to CEO and Lessons Learned Along the Way
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Nov 8, 2023

Orchestrate Unstructured Data Pipelines with Dagster and dlt

Load messy data sources into well-structured tables or datasets, through automatic schema inference and evolution.
Zaeem Athar
Name
Zaeem Athar
Handle
@zaeem
Oct 31, 2023

The Craft Of Open Source: a Flagsmith podcast

Pete Hunt discusses data orchestration, Dagster, and our onward journey.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Oct 31, 2023

Data Unlocked: How to Work Effectively With Your Data Teams

Nick Schrock on the relationship between data engineering and go-to-market.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Oct 20, 2023

CI/CD and Data Pipeline Automation (with Git)

Learn how to automate data pipelines and deployments by integrating Git and CI/CD in our Python for data engineering ...
Elliot Gunn
Name
Elliot Gunn
Handle
@elliot
Oct 19, 2023

The Tech Trek Podcast: Open source data orchestration

Pete Hunt shares insights on the challenges in the data orchestration market, and why Dagster is open-source.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Oct 13, 2023

Introducing Dagster Pipes

A new protocol and toolkit for integrating and launching compute into remote execution environments from Dagster.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Oct 13, 2023

Introducing External Assets

Use Dagster’s External Assets feature for data observability, lineage, data quality, and cataloging while bringing your ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Oct 12, 2023

Stop Reinventing Orchestration: Embedded ELT in the Orchestrator

Solve data ingestion issues with Dagster's Embedded ELT feature, a lightweight embedded library.
Pedram Navid
Name
Pedram Navid
Handle
@pdrmnvd
Oct 11, 2023

Improving the Dagster learning curve

Learn Dagster essentials and build asset-based data pipelines with Dagster University, our new self-guided course for ...
Erin Cochran
Name
Erin Cochran
Handle
Oct 10, 2023

Improving visibility into data operations with Dagster Insights

Gain operational observability on your data pipelines and bring cloud costs back under control with the Dagster ...
Jarred Colli
Name
Jarred Colli
Handle
@jarred
Oct 9, 2023

Introducing Asset Checks

Deliver high-quality data with Dagster Asset Checks, the ability to embed data quality checks into your data pipeline.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Johann Miller
Name
Johann Miller
Handle
@johann
Oct 4, 2023

The Orchestration Layer as the Data Platform Control Plane

Nick Schrock, founder and CTO of Dagster Labs, on The Data Stack Show.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Oct 2, 2023

Dagster 1.5: How Will I Know?

Ahead of Launch Week, we are proud to be rolling out some exciting new capabilities.
Yuhan Luo
Name
Yuhan Luo
Handle
@yuhan
Sep 29, 2023

Write-Audit-Publish in data pipelines

We look at the write-audit-publish software design pattern used in ETL to ensure quality and reliability in data ...
Elliot Gunn
Name
Elliot Gunn
Handle
@elliot
Sep 28, 2023

Escaping the Modern Data Trap

Launch Week kicks off October 9th with new functionality being shared each day. Our theme: Escaping the Modern Data ...
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Sep 21, 2023

Bringing Great Developer Experience to Data Teams

Nick Schrock on how Dagster is bringing software engineering principles to the data space, and what a great developer ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Sep 20, 2023

Pedram Navid: Why I Joined Dagster Labs

It is not every day you get to join a company working on building a product purpose-built for you.
Pedram Navid
Name
Pedram Navid
Handle
@pdrmnvd
Sep 14, 2023

A Dagster-Powered Spam Filter

Using Dagster, you can maintain data trust and protect the integrity of any user-generated service with this powerful ...
James Timmins
Name
James Timmins
Handle
@jamestimmins
Sep 13, 2023

Code Story

Pete Hunt joins Noah Labhart - startup founder & CTO - to discuss the origin story of Dagster.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Sep 10, 2023

Data Orchestration in an Increasingly Complex Data Ecosystem

Nick Schrock shares his perspective on the state of data orchestration technology and its application to help inform ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Sep 4, 2023

Factory Patterns in Python

We explore design patterns — reusable solutions to common problems in software design — as used in data engineering, ...
Elliot Gunn
Name
Elliot Gunn
Handle
@elliot
Aug 29, 2023

Migrating off dbt Cloud™

Looking for an alternative tool to orchestrate your dbt projects? Here’s a step-by-step guide to migrating from dbt ...
Tim Castillo
Name
Tim Castillo
Handle
@tims_tangents
Claire Lin
Name
Claire Lin
Handle
Aug 28, 2023

The Breakthrough Hiring Show with Pete Hunt

Pete and host James Mackey discuss strategic hiring for startups and the dangers of getting too big too fast.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Aug 28, 2023

ML pipelines for fine-tuning LLMs

LLM fine-tuning best practices for creating a clean production ML pipeline, streamlining model training, and ...
Odette Harary
Name
Odette Harary
Handle
@odette
Aug 24, 2023

The Happy Engineer Podcast: Engineering Hard Choices

Pete Hunt shares insights on building and leading a data engineering team and making hard engineering calls.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Aug 24, 2023

Adventures in DevOps

Testing and Development in the Data Domain
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Aug 21, 2023

Introducing Dagster Labs

In the spirit of simplification, the company formerly known as Elementl is now doing business as Dagster Labs.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Aug 18, 2023

Building an Outbound Reporting Pipeline

Learn how to use data engineering patterns and Dagster’s dynamic partitioning to build an outbound email report ...
James Timmins
Name
James Timmins
Handle
@jamestimmins
Aug 14, 2023

Parallel Computing on Dagster with Dask

Orchestrate your Dask computations and make your pipelines faster for larger data engineering and machine learning ...
Odette Harary
Name
Odette Harary
Handle
@odette
Aug 11, 2023

Type Hinting in Python

In part VI of our Data Engineering with Python series, we explore type hinting functions and classes, and how type ...
Elliot Gunn
Name
Elliot Gunn
Handle
@elliot
Aug 7, 2023

Environment Variables in Python

In part V of our series on Data Engineering with Python, we cover best practices for managing environment variables in ...
Elliot Gunn
Name
Elliot Gunn
Handle
@elliot
Aug 3, 2023

Whats New in Data

Data Orchestration, Dagster, and parallels to React.js
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Aug 3, 2023

Drill to Detail: Dagster, Orchestration and Software-Defined Assets

Dagster Labs founder Nick Shrock is interviewed by Rittman Analytics founder Mark Rittman
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Aug 2, 2023

The Scale Up Show: Interview with Pete Hunt

Ryan Staley interviewed Pete Hunt on how his experience at Facebook and Twitter is guiding his leadership of Dagster.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Aug 1, 2023

Orchestrating dbt™ with Dagster

Orchestrate dbt with Dagster’s popular dbt integration, now with major enhancements to supercharge your dbt models as ...
Rex Ledesma
Name
Rex Ledesma
Handle
@_rexledesma
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Jul 31, 2023

Speeding up the dbt™ docs by 20x with React Server Components

dbt docs slow? See how we dropped page load time and memory usage for a large dbt project by 20x using React Server ...
Marco Salazar
Name
Marco Salazar
Handle
@BkOptimism
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Jul 24, 2023

A Geek Leader: Interview with Pete Hunt

John Rouda interviewed Pete Hunt, CEO of Dagster Labs, on React.js, open source and data orchestration.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Jul 21, 2023

Dagster 1.4: Material Girl

The latest release brings major new dbt capabilities, new asset materialization controls, and more.
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Jul 6, 2023

Asset-Based Data Orchestration (from Data + AI Summit)

An overview of Dagster's asset-based orchestration approach, with data freshness sensors to trigger pipelines.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Jul 5, 2023

LLM training pipelines with Langchain, Airbyte, and Dagster

This tutorial shows you how to combine Langchain, Airbyte, and Dagster to build maintainable and scalable pipelines for ...
Jun 26, 2023

Introducing Two New Self-Serve Plans for Dagster Cloud

'Solo' and 'Team' plans, with event-based pricing, will replace the old compute-duration based plan. We explain why we ...
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Jun 22, 2023

Revisiting the Poor Man’s Data Lake with MotherDuck

See how much easier you can collaborate using DuckDB’s high-powered cloud version MotherDuck to build a one-system data ...
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Jun 15, 2023

The Dagster Master Plan

Elementl CEO Pete Hunt shares the three priorities that guide how we will evolve Dagster.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Jun 6, 2023

Backfills in Data & Machine Learning: A Primer

A step-by-step guide to using backfills and partitions to make data management more simple for data & ML engineers.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
May 31, 2023

Data Platform Podcast: Orchestration & Psychology featuring Pete Hunt

Jason and Iva are joined by Pete Hunt, CEO of Elementl, to discuss orchestration tools and the psychology of companies.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
May 24, 2023

Elementl Raises $33 Million in Series B Funding to Accelerate Data Orchestration and Unleash Advanced Data Use Cases

The new capital will accelerate the development and adoption of Dagster, the open-source, cloud-native data ...
May 24, 2023

Dagster and the Decade of Data Engineering

We are pleased to announce Elementl's $33M Series B and share our vision for what's next for Dagster and the practice ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
May 23, 2023

Building Better Analytics Pipelines

A recap of our live event on the benefits and techniques for orchestrating analytics pipelines.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Yuhan Luo
Name
Yuhan Luo
Handle
@yuhan
May 19, 2023

Introducing Dynamic Definitions for Flexible Asset Partitioning

Dagster’s dynamic partition definitions allow engineers to use the power of partitions in a broader range of scenarios.
Claire Lin
Name
Claire Lin
Handle
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
May 17, 2023

Deciphering Arcane Kubernetes and ECS Errors with Dagster

Recent enhancements allow Dagster to surface clearer and more actionable errors to accelerate your development cycles.
Daniel Gibson
Name
Daniel Gibson
Handle
May 16, 2023

Config Systems: Airflow and Dagster

Contrasting the Airflow and Dagster configuration systems by rewriting the Airflow Slack Integration.
Joe Van Drunen
Name
Joe Van Drunen
Handle
May 9, 2023

How to Maintain High Product & Code Quality As Your Startup Scales

Raising the quality bar requires process adjustments and a cultural shift.
Bosmat Eldar
Name
Bosmat Eldar
Handle
@bosmat
Apr 26, 2023

Dagster 1.3: Smooth Operator

Dagster 1.3 officially inducts Pythonic Config and Resources and brings new enhancements to Software-Defined Assets, ...
Yuhan Luo
Name
Yuhan Luo
Handle
@yuhan
Apr 21, 2023

Catalyst Cooperative: Liberating Public Utility Data with Dagster

The PUDL Project cleans and distributes analysis-ready energy system data to climate advocates, researchers, ...
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Apr 14, 2023

From Python Projects to Dagster Pipelines

In part IV of our series, we explore setting up a Dagster project, and the key concept of Data Assets.
Elliot Gunn
Name
Elliot Gunn
Handle
@elliot
Apr 10, 2023

Enabling Large-scale, Multi-cloud Computing with Dagster

Abstracting away infrastructure concerns in large-scale computing with conditional multi-cloud processing.
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Apr 4, 2023

Orchestrate Meltano Jobs with Dagster

Meltano provides 550 connectors and tools, all of which can be configured and orchestrated straight from Dagster.
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Apr 3, 2023

Community Memo: Pythonic Config and Resources

Major ergonomic improvements are coming to Dagster's config and resources systems, including a Pydantic frontend.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Ben Pankow
Name
Ben Pankow
Handle
Mar 21, 2023

Best Practices in Structuring Python Projects

We cover 9 best practices and examples on structuring your Python projects for collaboration and productivity.
Elliot Gunn
Name
Elliot Gunn
Handle
@elliot
Mar 20, 2023

Partitions in Data Pipelines

Partitioning is a technique that helps data engineers and ML engineers organize data and the computations that produce ...
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Mar 16, 2023

Tracking the Fake GitHub Star Black Market with Dagster, dbt and BigQuery

It's easy for an open-source project to buy fake GitHub stars. We share two approaches for detecting them.
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Yuhan Luo
Name
Yuhan Luo
Handle
@yuhan
Mar 9, 2023

Dagster 1.2: Formation

Enhanced partitioned asset support and the introduction of Pythonic config and resources, and integration updates.
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Mar 7, 2023

How We Deploy 5X Faster with Warm Docker Containers

Using pex, Serverless Dagster Cloud now deploys 4 to 5 times faster by avoiding the overhead of building and launching ...
Shalabh Chaturvedi
Name
Shalabh Chaturvedi
Handle
Mar 6, 2023

Python Packages: a Primer for Data People (part 2 of 2)

An introduction to managing Python dependencies and some virtual environment best practices.
Elliot Gunn
Name
Elliot Gunn
Handle
@elliot
Mar 6, 2023

Python Packages: a Primer for Data People (part 1 of 2)

The foundation of a solid Python project is mastering modules, packages and imports.
Elliot Gunn
Name
Elliot Gunn
Handle
@elliot
Feb 28, 2023

Dagster Integrations Update

Dagster offers 47 integrations to accelerate your development, and we are working hard to expand and enhance them.
Rex Ledesma
Name
Rex Ledesma
Handle
@_rexledesma
Feb 8, 2023

Migrating from Airflow to Dagster is now a Breeze

The newly released `dagster-airflow` library has made migrating off legacy Airflow and onto Dagster much easier.
Joe Van Drunen
Name
Joe Van Drunen
Handle
Jan 9, 2023

Build a GitHub Support Bot with GPT3, LangChain, and Python

In this tutorial, we tap into the power of OpenAI's ChatGPT to build a GitHub support bot using GPT3, LangChain, and ...
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Dec 22, 2022

Converting an ETL Script to Software-Defined Assets

Lets talk about moving from an ETL script to a robust Dagster pipeline using Software-Defined Assets.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Dec 16, 2022

Bringing Declarative Scheduling to dbt with Dagster

Declarative Scheduling takes the orchestration of dbt models as part of a larger pipeline to an entirely new level.
Sean Lopp
Name
Sean Lopp
Handle
@lopp
Dec 14, 2022

Dagster 1.1: Thank U, Next

A major release with Declarative Scheduling, multi-asset scheduling, and SDA partitioning. Plus Secrets management, ...
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Dec 8, 2022

Declarative Scheduling for Data Assets

Keep data assets up-to-date and determine whether source data has changed with declarative asset-based scheduling.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Dec 7, 2022

Evaluating Dagster for Better Skiing - and a New Job

How quickstart projects snowball into new careers. A common data PoC walkthrough with Dagster.
Sean Lopp
Name
Sean Lopp
Handle
@lopp
Dec 1, 2022

Build More Reliable Machine Learning Systems

Sandy Ryza explains how his background in machine learning has informed his work on the Dagster project.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Nov 30, 2022

Getting Stuff Done: a Guide to Productive Software Engineering

To be a more productive software engineer you need to master changes, how these affect the program and others on the ...
Alex Langenfeld
Name
Alex Langenfeld
Handle
@alex_langenfeld
Nov 21, 2022

Safe and Easy: Managing Secrets in Dagster Cloud

Dagster Cloud’s new Environment Variables UI makes it easy to set up scoped environment variables.
Erin Cochran
Name
Erin Cochran
Handle
Daniel Gibson
Name
Daniel Gibson
Handle
Nov 18, 2022

My Path to Elementl - Part 2

Pete Hunt takes over as CEO as Nick Schrock takes on the CTO role.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Nov 11, 2022

Pushing REST-API data to Google Sheets with Dagster

A total beginners tutorial in which we store REST API data in Google Sheets and learn some key abstractions.
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Nov 7, 2022

Adding Types to a Large Python Codebase

What we learned when we introduced dynamically typed code to a large Python codebase, bringing Dagster's public API to ...
Sean Mackesey
Name
Sean Mackesey
Handle
Oct 31, 2022

Orchestrating Machine Learning Pipelines with Dagster

How to use Dagster’s open source data orchestrator to build machine learning pipelines and train ML models.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Oct 27, 2022

Orchestrating Data Science at Zephyr AI

Zephyr AI applies data science to massive datasets of DNA and healthcare records to deliver novel AI-driven insights.
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Oct 25, 2022

Build a poor man’s data lake from scratch with DuckDB

DuckDB is so hot right now. Learn how to build a data lake from dbt using DuckDB for SQL transformations, along with ...
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Oct 19, 2022

The Unreasonable Effectiveness of Data Pipeline Smoke Tests

Data practitioners waste time writing unit tests to catch bugs they could have caught with smoke tests.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Oct 17, 2022

Web Workers are not the Answer

A tale of overstretched logs, counterintuitive web worker behavior, and ultimately a troublesome cursor issue.
Marco Salazar
Name
Marco Salazar
Handle
@BkOptimism
Alex Langenfeld
Name
Alex Langenfeld
Handle
@alex_langenfeld
Oct 16, 2022

Dagster at all 5 Steps of the Development Lifecycle

Dagster facilitates a data engineers work across all five steps in the development lifecycle.
Oct 6, 2022

A Dagster Crash Course

If you are looking to get up and running with Dagster in 10 minutes or less, this is a good place to start. Buckle up.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Oct 4, 2022

Postgres: a Better Message Queue than Kafka?

When lots of event logs must be stored and indexed, Kafka is the obvious choice. Naturally, our queue runs on Postgres.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Sep 20, 2022

Dagster vs. Airflow

Looking for an Apache Airflow alternative? See why data teams choose Dagster for data orchestration in this five-part ...
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Aug 24, 2022

How EvolutionIQ Rebuilt its ML Platform for Enormous Productivity.

A guide for CIOs/CTOs and engineering leaders looking to master the Modern Data Stack and develop a high performance ...
Fraser Marlow
Name
Fraser Marlow
Handle
@frasermarlow
Aug 17, 2022

Spend Less Time Debugging with Dagster

It’s not uncommon for a data engineer to devote 80% of their day to debugging. Dagster radically improves on this.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Owen Kephart
Name
Owen Kephart
Handle
Aug 9, 2022

Launching Dagster Cloud to GA

The enterprise orchestration platform that puts developer experience first: hybrid or serverless deployments, native ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Aug 5, 2022

Introducing Dagster 1.0: Hello

Announcing Dagster 1.0. - a stable foundation for building the orchestration layer for modern data platforms.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Aug 3, 2022

The Open Core Business Model

The relationship between Dagster, the open-source project, and Dagster Cloud, our hosted SaaS platform.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Jul 26, 2022

Dagster Cloud goes SOC 2

Elementl, the company behind the Dagster data orchestration tool achieves SOC2 compliance.
Selina Li
Name
Selina Li
Handle
Jul 25, 2022

Dagster Day: Announcing Dagster 1.0 and Dagster Cloud

The release of Dagster 1.0 and the GA launch of Dagster Cloud represent major milestones in the evolution of our ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Jul 12, 2022

Roman Roads in Data Engineering: Don't Write Data Pipelines from Scratch

Work in a way that lays the foundation for your next data product while you're building your current one.
Claire Lin
Name
Claire Lin
Handle
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Jun 23, 2022

The Data Exchange: Software-defined Assets

Nick Schrock on software-defined assets, a new approach to managing, maintaining, and orchestrating data declaratively.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Jun 22, 2022

My Path to Elementl: Pete Hunt

Pete Hunt discusses what caused him to make the leap from Twitter to Elementl.
Pete Hunt
Name
Pete Hunt
Handle
@floydophone
Jun 20, 2022

Orchestrating Python and dbt with Dagster

How asset-focused orchestration bridges the gap between some of data's most popular tools.
Owen Kephart
Name
Owen Kephart
Handle
Jun 15, 2022

Dagster 0.15.0: Cool for the Summer

In 0.15.0, software-defined assets are now marked fully stable and are ready for primetime.
Mollie Pettit
Name
Mollie Pettit
Handle
Mar 9, 2022

New in 0.14.0: Dagster-Airbyte Integration

0.14.0 introduces a deep integration with Airbyte: view Airbyte logs directly in Dagit, and every updated table will be ...
Owen Kephart
Name
Owen Kephart
Handle
Mar 1, 2022

Introducing Software-Defined Assets

Software-Defined Assets are a new abstraction that allows data teams to focus on the end products, not just the ...
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Mar 1, 2022

Dagster 0.14.0: Table Schema API + Pandera Integration

Introducing two asset observability-enhancing features: Table Schema API, and an integration with the dataframe ...
Sean Mackesey
Name
Sean Mackesey
Handle
Mar 1, 2022

Dagster 0.14.0: Never Felt Like This Before

We’re thrilled to release version 0.14.0 of Dagster. This version introduces much more mature version of ...
Mollie Pettit
Name
Mollie Pettit
Handle
Feb 17, 2022

Rebundling the Data Platform

'The Unbundling of Airflow' argued that modern data stack solutions (data ingestion, data transformation, reverse ETL) ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Dec 2, 2021

Introducing Dagster Cloud

Dagster Cloud, the enterprise orchestration platform that puts developer experience first, with fully serverless or ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Nov 20, 2021

Laying the Foundation of your Data Platform for the Era of Big Complexity

Listen to founder and CEO Nick Schrock talk about how Dagster helps tame the complexity and scale when working with ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Nov 17, 2021

Hello Big Complexity: Is Your Modern Data Stack Ready?

Listen to Nick Schrock discuss the evolution of data from Big Data to Big Complexity in this episode of the Mad Data ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Nov 16, 2021

Why Elementl and Dagster: The Decade of Data

Announcing our $14M Series A led by Index Ventures, alongside Sequoia Capital, Slow Ventures, Coatue, Amplify Partners, ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Nov 8, 2021

New in Dagster 0.13.0: Logging Improvements!

Logging without context, instance-wide handlers, capturing python logs, and more! Learn about the improvements we've ...
Owen Kephart
Name
Owen Kephart
Handle
Oct 28, 2021

Dagster 0.13.0: A New Foundation

We’re proud to announce 0.13.0 of Dagster with dramatic improvements to our core APIs, completely revamped UI, and ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Aug 10, 2021

Community Memo: Moving Dagster's Core APIs Towards 1.0

Dagster commits to a stable set of production-ready APIs for building solid data platforms.
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Jul 19, 2021

Dagster 0.12.0: Into the Groove

In 0.12.0, we introduce pipeline failure sensors, solid-level retries, and more convenient testing APIs.
Owen Kephart
Name
Owen Kephart
Handle
May 25, 2021

Community Memo: Approachability Improvements

In the last two months, we've made a set of changes aimed at making Dagster more approachable: to smooth out its ...
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
May 18, 2021

Incrementally Adopting Dagster at Mapbox

At Mapbox, we've adopted Dagster without breaking compatibility with our legacy Airflow systems -- and with huge gains ...
Ben Pleasanton
Name
Ben Pleasanton
Handle
May 13, 2021

Moving past Airflow: Why Dagster is the Next-generation Data Orchestrator

A comparison between Dagster and Airflow. Here we detail the differences between the two systems, and make the case for ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Apr 1, 2021

Dagster 0.11.0: Lucky Star

In 0.11.0, we introduce dynamic orchestration, a new backfill UI, and support for tracking asset lineage.
Mar 15, 2021

Building Shared Spaces for Data Teams at Drizly

Our small data infrastructure team built a data platform that supports users with different skillsets, letting anyone ...
Dennis Hume
Name
Dennis Hume
Handle
Jan 19, 2021

Dagster 0.10.0: The Edge of Glory

In 0.10.0, we introduce unique event-based scheduling capabilities, hardened deployments on Kubernetes, and new ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Max Gasner
Name
Max Gasner
Handle
Dec 9, 2020

Good Data at Good Eggs: Using Dagster to Manage the Data Platform

Running pipelines is only part of running a data platform. We need to manage the platform and control technical debt. ...
David Wallace
Name
David Wallace
Handle
@davidjwallace
Nov 5, 2020

Good Data at Good Eggs: Data Observability with the Asset Catalog

Dagster gives us a single "pane of glass" for data assets. Analysts can look up when a Stitch raw data ingest occurred, ...
David Wallace
Name
David Wallace
Handle
@davidjwallace
Oct 29, 2020

Dagster and dbt: Better Together

People sometimes ask us — should I use Dagster, or should I use dbt? We view Dagster and dbt as complementary ...
AJ Nadel
Name
AJ Nadel
Handle
@AJ_Nadel
Bob Chen
Name
Bob Chen
Handle
Oct 1, 2020

Good Data at Good Eggs: Data Infrastructure Correctness and Reliability

Dagster’s custom data types helped achieve correctness and reliability in our data ingest process, less downstream ...
David Wallace
Name
David Wallace
Handle
@davidjwallace
Oct 1, 2020

Good Data at Good Eggs: Part 1 of 4

Adopting Dagster transformed our data platform team. We hope our experience is encouraging to other teams facing ...
David Wallace
Name
David Wallace
Handle
@davidjwallace
Sep 16, 2020

Testing and Deploying PySpark Jobs with Dagster

Spark has a beautiful API but developing with it is a pain because different stages of development and deployment ...
Sandy Ryza
Name
Sandy Ryza
Handle
@s_ryz
Sep 15, 2020

Community Memo: September 2020 Update

A retrospective of our 0.9.0 release, a preview of our 0.10.0 roadmap, and Prezi's journey from a homegrown ...
Sep 10, 2020

Great Expectations for Dagster

We’re thrilled to announce a new integration between Dagster and a fellow open-source project, Great Expectations (GX).
Leor Fishman
Name
Leor Fishman
Handle
Aug 25, 2020

Forward Thinking Leaders

Nick Schrock shares insights on how to on how to sell new tech concepts to developers.
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Aug 11, 2020

Dagster: The Data Orchestrator

As a workflow engine, Dagster moves beyond ordering and executing data computations. It introduces a new primitive: a ...
Nick Schrock
Name
Nick Schrock
Handle
@schrockn
Max Gasner
Name
Max Gasner
Handle
Feb 26, 2020

Dagster 0.7.0: Waiting To Exhale

With 0.7.0 we set out improve the Dagster experience with large, production-scale pipelines, deployable to Kubernetes.
Oct 10, 2019

Dagster 0.6.0: Impossible Princess

Dagster 0.6.0 comes “batteries-included” and pluggable options to execute, monitor, schedule, deploy, and debug your ...
Jul 8, 2019

Introducing Dagster

Elementl announces an early release of Dagster, an open-source library for building ETL processes, ML pipelines and ...