Deep Dive: Platform Engineering with Dagster at Empirico

Data-driven biotechnology companies face unique big data engineering challenges that most organizations never encounter: orchestrating massive computational workloads across genetic datasets from millions of individuals, managing dynamic fan-out patterns for drug discovery pipelines, and maintaining cost visibility across multiple external platforms where compute expenses can spiral out of control.

In this deep dive, Empirico's Lead Platform Engineer Zach Romer will walk through the real-world orchestration patterns that enable life sciences companies to scale their data operations efficiently. He'll demonstrate how Dagster's graph-asset architecture handles dynamic fan-out execution across external platforms, enabling Empirico to process massive datasets while maintaining precise control over computational resources.

Zach will showcase practical strategies for big data orchestration, including how Dagster's asset-centric approach enables consistent tagging across platforms—critical for achieving cost transparency when compute workloads span AWS, specialized genomics platforms, and ML infrastructure. You'll see how this unified orchestration approach transforms cost management from reactive monitoring to proactive optimization, enabling teams to make data-driven decisions about computational resource allocation across complex, multi-platform genomics workflows.