Stop managing prompt spaghetti and start building production-ready LLM applications. In this deep dive, learn how DSPy's declarative framework eliminates manual prompt engineering while Dagster provides the orchestration and observability your AI systems need in production. We'll walk through a real-world example using the NY Times Connections puzzle, demonstrating automatic prompt optimization, model evaluation workflows, and how to move from proof-of-concept to production-grade LLM pipeline
Transform your LLM development from chaotic prompt strings to production-ready, self-optimizing pipelines.
In this Dagster Deep Dive, Alex Noonan and Colton Padden demonstrate how to escape the "prompt spaghetti" trap that plagues most LLM applications. Learn how combining DSPy's declarative framework with Dagster's orchestration creates maintainable, testable, and automatically optimized AI systems.
What You'll Learn
Why Traditional LLM Development FailsThe problems with hard-coded prompts and how they fall outside the software development lifecycle. Why LLM applications need proper orchestration just like data pipelines.
DSPy FundamentalsSignatures that define what you want without specifying how. Modules as building blocks for language model calls. Optimizers for automatic prompt tuning. Type-safe inputs and outputs using Pydantic models.
Dagster for ProductionAsset-centric orchestration for LLM pipelines with built-in observability and metadata tracking. Dagster Components for declarative pipeline setup. Moving models from development to production.
Real-World Demo: NY Times Connections PuzzleComplete walkthrough using actual game data. Model evaluation and optimization workflows comparing performance across different LLM providers.
.jpg)
Stop managing prompt spaghetti and start building production-ready LLM applications. In this deep dive, learn how DSPy's declarative framework eliminates manual prompt engineering while Dagster provides the orchestration and observability your AI systems need in production. We'll walk through a real-world example using the NY Times Connections puzzle, demonstrating automatic prompt optimization, model evaluation workflows, and how to move from proof-of-concept to production-grade LLM pipeline
Transform your LLM development from chaotic prompt strings to production-ready, self-optimizing pipelines.
In this Dagster Deep Dive, Alex Noonan and Colton Padden demonstrate how to escape the "prompt spaghetti" trap that plagues most LLM applications. Learn how combining DSPy's declarative framework with Dagster's orchestration creates maintainable, testable, and automatically optimized AI systems.
What You'll Learn
Why Traditional LLM Development FailsThe problems with hard-coded prompts and how they fall outside the software development lifecycle. Why LLM applications need proper orchestration just like data pipelines.
DSPy FundamentalsSignatures that define what you want without specifying how. Modules as building blocks for language model calls. Optimizers for automatic prompt tuning. Type-safe inputs and outputs using Pydantic models.
Dagster for ProductionAsset-centric orchestration for LLM pipelines with built-in observability and metadata tracking. Dagster Components for declarative pipeline setup. Moving models from development to production.
Real-World Demo: NY Times Connections PuzzleComplete walkthrough using actual game data. Model evaluation and optimization workflows comparing performance across different LLM providers.
