Dagster 1.13: Octopus's Garden

Dagster 1.13 is about making the framework easier to adopt: faster to prototype with, easier to scale, and easier to use from both code and AI tooling.

This release is centered on a few themes:

• Better support for working with AI tools and coding agents.
• Better ways to model and operate partitioned data.
• Expanding our library of Components, and making state-backed components enabled by default, making it easier to work with integrations built on external metadata.
• Preview support for virtual assets are here, making it easy to represent logical assets like database views.

Read on for the highlights.

Better support for AI-assisted Dagster development

In Dagster 1.13, we’re making it easier to build with modern AI tooling.

We released the open-source dagster-io/skills repository: a collection of Dagster-focused AI skills designed for tools like OpenAI Codex, Claude Code, OpenCode, other LLM harnesses. Instead of binding Dagster guidance to one assistant product, the skills package exposes Dagster-specific knowledge into a reusable format that can travel across agentic coding tools.

That makes them useful across the full lifecycle of a Dagster project:

• Prototyping a new project and definitions
• Building production-grade systems with the right patterns for assets, automation, structure, and deployment
• Troubleshooting failures, run behavior, and configuration issues

You can quickly install these skills via npx skills or through Claude’s plugin marketplace:

npx skills add dagster-io/skills
# or install directly in Claude Code
/plugin marketplace add dagster-io/skills
/plugin install dagster-expert@dagster-skills
# then ask your coding agent something like:
"Help me debug why my materialization failed"

If you are already using AI tools to help build and operate data platforms, this release makes Dagster a much better fit for that workflow.

Partitioned asset checks are here!

Support for partition-aware asset checks have landed, making it so checks can now evaluate a specific partition of an upstream asset instead of the whole dataset. For teams working with time-based and other partitioned assets, this closes an important gap: you can now express data quality and validation logic at the same granularity as the assets themselves.

In practice, this means teams can now write checks that line up cleanly with how partitioned data is actually produced and monitored:

import dagster as dg

daily_partitions = dg.DailyPartitionsDefinition(start_date="2025-01-01")

@dg.asset(partitions_def=daily_partitions)
def daily_orders(context: dg.AssetExecutionContext):
  partition_key = context.partition_key
  return load_orders_for_date(partition_key)
  
@dg.asset_check(
  asset=daily_orders,
  description="daily orders passed validation",
  partitions_def=daily_partitions,
)
def daily_orders_check(daily_orders) -> dg.AssetCheckResult:
  return dg.AssetCheckResult(
    passed=len(daily_orders) > 0,
    metadata={"row_count": len(daily_orders)},
  )

This release also rounds out the rest of the partition workflow, selection, and partition-aware execution feel much more natural in day-to-day use.

A stronger developer experience with dg

The dg CLI and related deployment workflows picked up a substantial set of quality-of-life improvements. But the bigger story is how we think about CLI tooling in an AI-native world.

We increasingly see command-line utilities as a strong abstraction layer for AI systems: they are explicit, composable, scriptable, and much easier for agents to call reliably than ad hoc product surfaces. We continued leaning into that idea by expanding dg api so that more of Dagster's interaction surface is available through structured commands.

That means agents (or humans!) can more cleanly inspect and work with assets, runs, schedules, sensors, deployments, secrets, and other parts of the Dagster platform through a stable interface built for programmatic use.

For example, an agent can discover and inspect jobs through a straightforward CLI interface like this:

$ dg api job list
NAME      DESCRIPTION  SCHEDULES SENSORS ASSET JOB
my_daily_job  Daily snapshot 1     2    Yes
my_sensor_job Sensor-driven 0     1    No
$ dg api job get my_daily_job
Name:    my_daily_job
Description: Daily snapshot
Asset Job: Yes
Tags:    env=prod, team=data
Schedule:  daily (0 0 * * *) [RUNNING]
Sensor:   file_watcher [RUNNING]

For human users, that same investment pays off as clearer tooling and a more consistent operational surface. For teams using AI assistants, it means Dagster becomes much easier to inspect, operate, and troubleshoot from within the tools where they already work.

State-backed components are enabled by default

State-backed components are the framework Dagster uses for integrations whose definitions depend on external metadata. Instead of re-querying external systems every time a code location loads, Dagster can fetch that state at controlled times, persist it, and then build definitions from the cached result.

For data teams, that means a more predictable experience when working with integrations like dbt, Fivetran, Airbyte, Tableau, Looker, Sigma, Power BI, and others.

Starting in Dagster 1.13, persisted local state becomes the default for state-backed components. For these components, teams should refresh definition state as part of CI/CD so each deploy is built against fresh external metadata, or on some defined schedule. If you use the GitHub Actions workflow generated by dg plus deploy configure, that refresh step is included by default. See the documentation on managing state in CI/CD.

Preview support for virtual assets

We've also added an experimental new feature: virtual assets.

While Dagster allows you to model arbitrarily complex graphs of assets, historically it's been challenging to accurately represent and track assets that don't require explicit computation in order to incorporate changes from their parents.

The most common example is database views. You only need to execute the function of a view when you want to change its definition, and otherwise it automatically
reflects the latest data from its upstream tables whenever it's queried.

However, because Dagster had no way of distinguishing between this type of asset and a more typical one, you'd experience challenges with other parts of the system, such as Declarative Automation and stale status calculation, that assumed that the only way for the contents of an asset to update would be for a materialization event to occur.

Now, when you define an asset you can pass is_virtual=True to indicate that the asset's contents update automatically when its parents change:

@dg.asset(is_virtual=True, deps=[upstream_table])
def my_view(context: dg.AssetExecutionContext, db: MyDatabase):
	db.execute("CREATE OR REPLACE VIEW my_view AS SELECT * FROM upstream_table")

If you want your automation conditions to be aware of virtual assets in the dependency chain, you can use resolve_through_virtual() to look through them to their non-virtual ancestors:

dg.AutomationCondition.eager().resolve_through_virtual()

We’re also shipping with opt-in support for automatically treating all views defined in your dbt project as virtual assets:

type: dagster_dbt.DbtProjectComponent
attributes:
 translator:
  enable_dbt_views_as_virtual_assets: true

This feature is currently in preview, which means we do not recommend it for production use cases as we gather feedback and potentially make changes to the API and functionality. We’re excited to see how you use this!

Easier adoption with 20+ additional components

Dagster 1.13 continues the push toward declarative, integration-friendly pipeline building.

Across the 1.12.x line, we added 20+ additional components, making it easier to connect Dagster to the tools, warehouses, cloud services, and operational systems teams already rely on.

On the Components front, this cycle added or expanded support for:

• DbtCloudComponent for loading dbt Cloud projects as Dagster assets.
• Declarative pipeline support for Spark in feature preview.
• Azure resource components for Blob Storage and ADLS2.
• GCP resource components for BigQuery, GCS, GCS File Manager, and Dataproc.
• New component and asset-loading capabilities across Databricks, Tableau, Looker, Census, Polytomic, and more.

Several integrations also got deeper operational support:

• dbt_cloud_assets now supports partitioned assets.
• DatabricksAssetBundleComponent became more flexible, including job-level subsetting and better asset mapping.
• DatabricksWorkspaceComponent can automatically cancel Databricks jobs when a Dagster run is terminated.
• Fivetran gained a polling sensor for observability, richer connector metadata, retry-on-reschedule behavior, and resync support.
• BI integrations now enrich assets with more table and storage metadata automatically.

There is also a clear pattern across these changes: integrations are not just being added, they are becoming more observable, more configurable, and easier to fit into a consistent Dagster model. That lowers the barrier to getting started while giving teams room to grow into more sophisticated deployments.

One especially useful example is improved auto-matching based on table metadata. As more integrations emit dagster/table_name metadata automatically, Dagster can do a better job connecting assets that refer to the same underlying table even when their asset keys differ. For teams stitching together assets across ingestion tools, BI tools, warehouses, and code-defined transformations, that reduces manual wiring and makes cross-system lineage feel much more natural.

from dagster import TableMetadataSet

@dg.asset(
  key=dg.AssetKey(["raw", "customers"]),
  metadata={**TableMetadataSet(table_name="analytics.raw.customers")},
)
def raw_customers():
  ...
@dg.asset(
  deps=[
    dg.AssetDep(
      "upstream_table",
      metadata={**TableMetadataSet(table_name="analytics.raw.customers")},
    )
  ]
)
def customer_health_report():
  ...

In this case, Dagster can infer that customer_health_report depends on raw_customers because they point at the same table, even though the asset keys do not match.

Dagster+ improvements

Dagster+ also saw meaningful upgrades during this cycle.

These included organization-level timezone settings, service users for Pro accounts, more resilient code server redeploy behavior, improved agent behavior during deploys and failure recovery, and continued expansion of insights and alerting workflows.

For teams running Dagster+ in production, these changes improve both the operator experience and the quality of the observability surface area around jobs, code locations, and alerts.

Looking forward

With Dagster 1.13, data teams should be able to adopt Dagster faster and move faster once they do. From Dagster Skills and AI-assisted workflows, to partitioned asset checks, to 20+ additional components, this release is about making Dagster more useful in the work data engineers and analysts actually do every day.

Upgrade to the latest 1.13 release to try it out, and let us know what you build! 🚀

‍