Community Showcase Part 3

Some of our favorite Dagster use cases are the ones we never could have predicted.

People in the community are using Dagster to explore public datasets, monitor infrastructure, automate research workflows, build internal tools, and experiment with entirely new kinds of data applications. Some projects are deeply technical, some are wonderfully niche, and all of them reflect the creativity of the people building with Dagster.

This post highlights a few community-built projects that caught our attention along with the stories behind them, why their creators picked Dagster, and what the experience of building them was like.

Kevin Longe

Linkedin | Github

Tell us a little about yourself and what you work on

I'm the Head of Data at a sports analytics company based in the UK. I actually started my career as what was essentially an analyst, but I quickly found myself automating away a lot of the repetitive work. That naturally led me toward programming and data engineering.

Like a lot of people a few years ago, I initially thought I wanted to be a data scientist. When I landed my first role with that title, though, I realized most of the work was actually data engineering and it turned out that was a much better fit for me. I really enjoy building reliable systems and creating tooling that makes other engineers more productive.

Outside of my day job, I spend a lot of time building open source software. Lately that's been SQLBuild, a SQL transformation framework that reflects many of my own ideas about developer experience, testing, and orchestration. A big part of what motivates me is building tools that make iteration faster and allow people to solve problems with less friction.

How did you first discover Dagster?

I actually knew about Dagster before I ever used it. At a previous company we were running Airflow, so I was aware there was this newer orchestrator people were talking about, but I never had a chance to try it.

It wasn't until my next role that I inherited a Dagster deployment from the engineer I was replacing. After working with it for a while, I pretty quickly became a fan.

The thing that immediately stood out to me was Dagster's asset-first approach. Having lineage as a first-class concept just felt like the right abstraction for data engineering. Instead of thinking purely in terms of tasks and schedules, you think about the assets your business actually cares about and how they're related. That mental model clicked with me almost immediately.

After that experience, choosing Dagster for my current company wasn't really a difficult decision.

What project have you been building with Dagster?

The biggest project I've been working on is SQLBuild, which you can think of as an alternative SQL transformation framework in the same space as dbt.

The philosophy behind SQLBuild is centered around fast iteration loops and what I'd describe as a "low floor, high ceiling." I want people to be productive with as little configuration and complexity as possible, while still giving them powerful capabilities when they need them. That means staying stateless by default and letting users opt into additional complexity only where it provides real value.

One area I think data engineering has historically underinvested in is testing. SQLBuild makes integration-style and end-to-end testing first-class. You define mock source data, specify the expectations for the models you're interested in, and SQLBuild executes the entire dependency graph needed to validate that workflow. It feels much closer to how software engineers test applications than how data teams traditionally test pipelines.

Working with Dagster as a way to deploy SQLBuild, was surprisingly straightforward. Its Python APIs are incredibly easy to integrate with, and because assets are such a natural abstraction, things like translating SQLBuild's testing concepts into Dagster Asset Checks felt obvious.

What was the hardest or most interesting problem you solved?

SQL transformation frameworks are foundational infrastructure, so people understandably move cautiously. Nobody wants to rebuild the heart of their data platform unless they're confident it will be around for years and solve real problems. Building that trust takes time, and I think the best way to do that is by using SQLBuild in production myself.

From a technical perspective, though, the most challenging problem was implementing state-aware orchestration while keeping the system as transparent and stateless as possible.

One of the design goals was avoiding a dedicated state database where possible and instead relying on append-only state stored in the warehouse. That kept the architecture simpler for users, but it made the orchestration logic substantially harder to get right.

Interestingly, the Dagster integration itself wasn't the difficult part. Dagster exposes clean extension points through Python, so plugging SQLBuild into its asset model was relatively painless. Most of the hard engineering work lived inside SQLBuild itself.

Why was Dagster a good fit for this project?

The lineage capabilities are fantastic because they give users a clear understanding of how everything connects together. When you're building transformation frameworks, that visibility becomes incredibly valuable, not just for debugging, but for understanding the impact of changes before you make them.

The Asset Check abstraction was another really natural fit. SQLBuild already had concepts around audits, validation, and testing, and mapping those into Asset Checks meant users could see operational health alongside the assets themselves without inventing an entirely separate system.

What I also appreciate is that Dagster never feels like it's getting in your way. Even though SQLBuild has its own abstractions and workflow, once it's integrated it still feels like a native Dagster experience. Users get all the benefits of the Dagster UI, lineage visualization, scheduling, and observability without having to think about where one system ends and the other begins.

If I'm building a data platform today, Dagster is still the orchestrator I'd choose.

What advice would you give someone starting with Dagster?

The biggest piece of advice I'd give is to learn Dagster as it exists today rather than relying on older tutorials or examples. The platform has evolved a lot over the last few years, particularly around assets and the new dg CLI.

Start by understanding assets and lineage, get comfortable with the core abstractions, and then gradually introduce things like asset checks, automation, partitions, sensors, or more advanced orchestration patterns as your needs grow.

I generally believe that the best tools have a low floor and a high ceiling. Dagster fits that philosophy well. You can get productive very quickly without sacrificing the ability to build sophisticated systems later, and I think that's one of the reasons it's become my orchestrator of choice.

Boris Resnick

Linkedin | Github

Tell us a little about yourself and what you work on

I am a software architect and technical lead with a background in aerospace and 5G technology. Currently, I am the founder of Flyvercity, where we build drone fleet management systems and aviation infrastructure. Alongside this, I am architecting the backend infrastructure for embyad.com. My focus across both is building reliable, data-driven systems that bridge AI capabilities with deterministic orchestration.

How did you first discover Dagster?

I discovered Dagster while architecting a model training pipeline for embyad.com. We needed to build an automated machine learning pipeline that chains multiple heavy foundation and vision-language models to handle automated dataset labelling and custom computer vision model training. Standard workflow managers or basic script automation completely fell apart when handling the complex data dependencies, versioning, and state management required for this kind of cascaded AI infrastructure. I was searching for an orchestrator that treated data assets as first-class objects rather than just a sequence of tasks, and Dagster was the clear choice.

What project have you been building with Dagster?

I use Dagster across two completely different projects: embyad.com: I built an automated, end-to-end machine learning training pipeline. It handles everything from data ingestion and distributed multi-model inference for auto-labelling, to dataset construction, model training, and ONNX export for a real-time computer vision system. We scale this across GPU/CPU worker nodes using Dask, with Dagster orchestrating the entire lifecycle.

Flyvercity: An AI-enabled drone fleet management platform and aviation infrastructure. In this project, I use Dagster to orchestrate the processing of massive amounts of flight data and complex telemetry streams.

What was the hardest or most interesting problem you solved?

Orchestrating a highly branching, dynamic ML training pipeline and ensuring correct resource distribution across a GPU-enabled cluster. Because our auto-labelling process relies on a chain of different vision and language models, the data flow isn't a linear path - it branches dynamically based on the input data characteristics. The real challenge was mapping these complex, data-dependent execution graphs in Dagster while simultaneously managing the physical execution layer via Dask. Ensuring that heavy model inference stages were correctly routed to specific GPU worker nodes - without bottlenecking the CPU-bound data preprocessing or ingestion stages - required proper alignment between Dagster’s orchestration logic and the underlying cluster topology.

Why was Dagster a good fit for this project?

Both of these projects are pure Python, and Dagster fits into that ecosystem seamlessly. Its philosophy of software assets was the primary driver for us. The asset-centric model made it much easier to track data across our branching ML pipelines and flight data streams.

However, one limitation I hit early on was a lack of strict static validation at configuration and asset-loading time. To address this, I built and open-sourced dagster-static. It provides a way to enforce static validation and structured compile-time checks right when the asset definitions load. This ensures that misconfigurations or structural pipeline errors are caught immediately before any heavy orchestration or remote execution ever kicks off, giving a pure Python pipeline the type of safety boundaries you normally only get in strictly compiled environments.

What advice would you give someone starting with Dagster?

A full-blown Dagster deployment - with its code servers, multiple components, and multi-project configurations - can quickly become overwhelming. Don't build for eventual scale on day one. Start as simply and straightforwardly as possible using a basic local setup. Avoid introducing unnecessary infrastructure complexity until you actually hit the wall that requires it. Instead, focus your energy on keeping your pipeline code highly modular, clean, and well-structured. If your asset definitions and Python code are clean, refactoring and scaling your Dagster topology later becomes trivial.