Orchestration is More than Scheduling: Declarative Automation in Dagster

Modern data platforms aren't just collections of independent cron jobs; they are graphs of hundreds of interconnected assets, each with unique freshness requirements, partitioning schemes, and downstream consumers. The real challenge isn't deciding when code should run, it's deciding what actually needs to run.

Traditional schedulers are excellent at launching work on a fixed cadence, but they're less effective at continuously answering questions like:

• Which assets are stale?
• Which downstream assets require recomputation?
• Which partitions are ready to materialize?
• Which assets must wait for their dependencies?

As platforms grow, this orchestration logic often becomes scattered across schedules, sensors, cross-job triggers, and custom code, making systems increasingly difficult to understand and maintain.

Declare the goal, not the schedule

Dagster fully supports traditional orchestration patterns, including schedules and sensors, when they're the right tool for the job. Declarative Automation offers a different approach for problems that are better expressed as desired state.

Instead of imperatively describing when work should run, you declare what it means for an asset to be up to date. Dagster's automation daemon continuously evaluates those conditions against the live state of your asset graph, its dependencies, partitions, data versions, and execution state, and determines exactly what work needs to run.

Rather than hand-authoring an ever-growing collection of schedules and triggers, you describe the desired state of your data platform, and Dagster continuously works to bring it into that state.

Common automation policies, one declarative model

In Dagster, data pipelines are modeled as assets: datasets, tables, machine learning models, or other persistent outputs together with the code that produces them. Assets form a graph that captures how data flows through your platform and how each asset depends on the others.

Declarative Automation builds on that asset graph. You attach an automation condition directly to an asset, describing what it means for that asset to be considered up to date. Dagster continuously evaluates those conditions against the current state of the graph and requests work whenever an asset no longer satisfies its declared policy.

The built-in automation conditions cover many of the most common orchestration patterns:

import dagster as dg

# Update as soon as any upstream dependency changes — event-driven propagation.
@dg.asset(deps=["upstream"], automation_condition=dg.AutomationCondition.eager())
def eager_asset() -> None: ...

# Update once per cron tick, but only after all upstream deps have refreshed.
@dg.asset(deps=["upstream"], automation_condition=dg.AutomationCondition.on_cron("@hourly"))
def hourly_asset() -> None: ...

# Fill anything that's missing, without re-running what already exists.
@dg.asset(deps=["upstream"], automation_condition=dg.AutomationCondition.on_missing())
def backfill_asset() -> None: ...

These aren't separate systems; they are different ways to define an asset's 'up to date' state. For example, on_cron() isn't just a simple schedule. Each time the cron triggers, Dagster evaluates the asset graph state. It only requests the asset if upstream dependencies are ready; if not, it waits. Time becomes one input among many, including dependency state, partitions, and freshness.

Whether you prefer traditional schedules and sensors or choose Declarative Automation for part of your platform, both approaches work together within the same orchestration system. Declarative Automation simply gives you a higher-level way to express orchestration policies when your goal is to describe the desired state rather than manually coordinate execution.

Declarative automation shines with partitions

Partitioned assets are where Declarative Automation becomes especially powerful. In imperative orchestration systems, engineers often end up writing custom logic to answer questions like:

• Which partitions are ready to process?
• How do hourly partitions map to daily partitions?
• Which downstream partitions should run next?
• How do we avoid reprocessing work that's already complete?

With Declarative Automation, you don't encode that orchestration logic yourself. Instead, automation conditions are evaluated independently for each partition, and Dagster determines exactly which partitions need to be materialized.

import dagster as dg

@dg.asset(partitions_def=dg.HourlyPartitionsDefinition("2025-01-01-00:00"))
def upstream() -> None: ...

@dg.asset(
    deps=[upstream],
    automation_condition=dg.AutomationCondition.on_missing(),
    partitions_def=dg.DailyPartitionsDefinition("2025-01-01"),
)
def downstream() -> None: ...

Here, the upstream asset is partitioned hourly while the downstream asset is partitioned daily. Rather than manually tracking when all 24 hourly partitions for a given day have arrived, on_missing() simply declares the desired outcome: every daily partition should exist.

Dagster uses its understanding of the asset graph and partition mappings to determine when enough upstream data is available to produce each daily partition. Once that condition is met, it requests the partition exactly once.

You aren't describing how to coordinate hourly and daily partitions, just the desired state. This model scales seamlessly: whether you are dealing with differing partition granularities, dynamic partitions, or evolving dependency graphs, the same declarative logic applies without requiring extra orchestration code.

Composing automation policies

Real-world orchestration rarely comes down to a single trigger. An asset might need to refresh every five minutes, but only if new data has arrived. Or it might run immediately when an upstream dependency updates, unless another dependency is still materializing.

Rather than scattering this logic across multiple schedules, sensors, and trigger callbacks, Declarative Automation lets you express it as a single policy by composing automation conditions with logical operators.

import dagster as dg

5min_cron = dg.AutomationCondition.newly_updated().since(
    dg.AutomationCondition.on_cron("*/5 * * * *")
)

custom_condition = dg.AutomationCondition.on_cron("*/5 * * * *") | (
    dg.AutomationCondition.any_deps_updated()      # when any dependency updates
    & 5min_cron                                # and it changed since the last tick
    & ~dg.AutomationCondition.any_deps_missing()   # but not if deps are missing
    & ~dg.AutomationCondition.any_deps_in_progress()  # and not mid-run
)

@dg.asset(automation_condition=custom_condition, deps=["upstream_1", "upstream_2"])
def combined_asset(context: dg.AssetExecutionContext): ...

This policy can be read almost like English:

• Run every five minutes.
• Or run sooner if an upstream dependency has been updated since the last cron tick.
• But don't run if dependencies are missing or currently materializing.

The important idea isn't the specific expression, it's that the orchestration policy lives in one place alongside the asset it governs. Instead of coordinating multiple schedules and sensors that interact with one another, you define a single declarative expression that captures the desired behavior.

Sometimes, however, the unit of orchestration isn't an individual asset. You may want a collection of assets to execute together in a single run, for example, when generating a report or publishing a coordinated dataset.

For those cases, job-level automation conditions (currently in preview) apply the same declarative model to an entire asset job:

analytics_job = dg.define_asset_job(
    "analytics_job",
    selection=[processed_data, report],
   automation_condition=dg.AutomationCondition.all_job_root_assets_match(
        dg.AutomationCondition.eager()
    ),
)

When the wrapped condition becomes true for the job's root assets, Dagster launches a single run for the entire job. The policy itself doesn't change, you simply choose whether it should govern an individual asset or a larger unit of work.

Test your logic before it hits production

Automation policies are application logic, and like any other application logic, they should be tested before they're deployed.

evaluate_automation_conditions() lets you evaluate automation policies against an ephemeral Dagster instance, making it easy to verify exactly what work would be requested without running the automation daemon.

def test_eager_condition():
    instance = dg.DagsterInstance.ephemeral()

    result = dg.evaluate_automation_conditions(defs=[upstream, downstream], instance=instance)
    assert result.total_requested == 0

    dg.materialize([upstream], instance=instance)

    result = dg.evaluate_automation_conditions(
        defs=[upstream, downstream], instance=instance, cursor=result.cursor
    )
    assert result.total_requested == 1

This test verifies that materializing the upstream asset causes exactly one downstream asset to be requested.

The same approach works for more sophisticated policies, including partitioned assets, cron-based conditions, and composed expressions. Rather than waiting for a scheduler tick or deploying the automation daemon, you can evaluate your policies in isolation and verify that they behave as expected.

Because automation conditions are deterministic expressions over the state of your asset graph, they can be unit tested just like the rest of your application code. As your orchestration logic grows more sophisticated, having fast, repeatable tests makes it easier to evolve your automation policies with confidence.

A different approach to orchestration

Data platforms are dynamic; dependencies evolve and data arrives unpredictably, making imperative scheduling difficult. Declarative Automation solves this by focusing on the desired state of assets rather than execution paths. While traditional schedules remain useful for specific integrations, Declarative Automation provides a simpler, more resilient way to express orchestration policies. Instead of maintaining a complex network of triggers, you simply define what 'up to date' means and let Dagster handle the rest.