Meet Compass — Dagster’s new AI data analyst for Slack. Turn questions into trusted insights, instantly. → Try Compass now.

Blog

When (and When Not) to Optimize Data Pipelines
When (and When Not) to Optimize Data Pipelines

November 17, 2025

When (and When Not) to Optimize Data Pipelines

Engineers often optimize the wrong parts of their pipelines, here's a profiling-first framework to identify real bottlenecks and avoid the premature optimization trap.

Alex Noonan
Read more
All Blogs
Releases
Engineering
Product
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Sling Out Your ETL Provider with Embedded ELT
Sling Out Your ETL Provider with Embedded ELT

April 3, 2024

Sling Out Your ETL Provider with Embedded ELT

How we saved $40k and gained better control over our ingestion steps.

Nick Roach
Product
Exploring The Data Engineering Lifecycle
Exploring The Data Engineering Lifecycle

March 26, 2024

Exploring The Data Engineering Lifecycle

Learn the fundamentals of a healthy data engineering lifecycle to optimize pipeline and asset production.

Sandy Ryza
Engineering
New Dagster Integration: Include OpenAI Calls Into Your Data Pipelines
New Dagster Integration: Include OpenAI Calls Into Your Data Pipelines

March 11, 2024

New Dagster Integration: Include OpenAI Calls Into Your Data Pipelines

The new dagster-openai integration lets you tap into the power of LLMs in a cost-efficient way.

Yuhan Luo
Engineering
Breaking Packages in Python
Breaking Packages in Python

February 27, 2024

Breaking Packages in Python

An exposé of the nooks and crannies of Python’s modules and packages.

Pedram Navid
Engineering
Balancing the Data Scales: Centralization vs. Decentralization
Balancing the Data Scales: Centralization vs. Decentralization

February 23, 2024

Balancing the Data Scales: Centralization vs. Decentralization

Learn how organizations can harness the strengths of both approaches to optimize their data operations.

TéJaun RiChard
Engineering
Standardize Pipelines with Domain-Specific Languages
Standardize Pipelines with Domain-Specific Languages

February 8, 2024

Standardize Pipelines with Domain-Specific Languages

By implementing DSLs, data teams can open their data platform to many more users without compromising on standards.

Elliot Gunn
Engineering
Thinking in Assets When Building Data Pipelines
Thinking in Assets When Building Data Pipelines

February 5, 2024

Thinking in Assets When Building Data Pipelines

How to develop data pipelines using Software-defined Assets.

Tim Castillo
Engineering
What Dagster Believes About Data Platforms
What Dagster Believes About Data Platforms

January 29, 2024

What Dagster Believes About Data Platforms

The beliefs that organizations adopt about the way their data platforms should function influence their outcomes. Here are ours.

Sandy Ryza
Product
Announcing Dagster 1.6: Back to Black
Announcing Dagster 1.6: Back to Black

January 12, 2024

Announcing Dagster 1.6: Back to Black

Major UI enhancements, Dagster Pipes upgrades and of course, dark mode :-)

Sandy Ryza
Releases
Retain.ai joins Dagster Labs
Retain.ai joins Dagster Labs

January 10, 2024

Retain.ai joins Dagster Labs

We’re excited and humbled to bring the Retain.ai organization into our fold to help build out Dagster’s data orchestration capabilities.

Pete Hunt
Product
How Dagster Labs runs Dagster: Open-Sourcing our Own Pipelines
How Dagster Labs runs Dagster: Open-Sourcing our Own Pipelines

December 4, 2023

How Dagster Labs runs Dagster: Open-Sourcing our Own Pipelines

A technical deep dive into the patterns and implementations of the Dagster Open Platform using our open-sourced code and dbt models.

Tim Castillo
Engineering
Scaling Dagster’s DAG Visualization to Handle Tens of Thousands of Assets
Scaling Dagster’s DAG Visualization to Handle Tens of Thousands of Assets

November 29, 2023

Scaling Dagster’s DAG Visualization to Handle Tens of Thousands of Assets

How the Dagster frontend team rapidly scaled Dagster’s DAG visualization for enterprise-sized data asset graphs.

Jordan Sanders
Engineering
No results, please try different filters.

Dagster Newsletter

Get updates delivered to your inbox