Dagster 1.9: Declarative Automation + BI

Declarative automation has officially graduated, BI in your asset graph, Airlift to streamline migrations, and more.

The Dagster 1.9 release – named Spooky – is a big, perhaps even monstrous, one. It:

Marks declarative automation generally available and stable, and extends it to asset checks.
Introduces four integrations with business intelligence (BI) tools: Tableau, Power BI, Looker, and Sigma.
Introduces Airlift, a toolkit to accelerate migrating from Airflow to Dagster.
Makes changes to the UI information architecture to make important entities easier to find.
and more!

Read on for the details.

Automation

Stable declarative automation

Declarative automation is now marked stable! Dagster now offers a mature, first-class way to automatically materialize assets when user-provided criteria are met, such as waiting for new upstream data. It can be used by simply assigning AutomationConditions to individual assets. The declarative automation system, first introduced in Dagster 1.8, is the successor to the experimental auto-materialize system, and was designed to be more composable, more flexible, and easier to use.

We’ve also extended declarative automation in a couple of ways:

Asset checks can now be assigned AutomationConditions.
(Experimental) AutomationConditions can now be built from arbitrary Python functions. This allows defining custom automation logic that can’t be expressed using compositions of built-in automation conditions.

Learn more in the documentation.

Job-less sensors and schedules

We previously released the ability to create schedules and sensors without defining a job using the target argument. In 1.9, the target arguments are now marked stable.

Business intelligence (BI) tool integrations

We’ve added integrations between Dagster and some of the most widely used business intelligence (BI) tools: Looker, Power BI, Tableau, and Sigma.

Data assets orchestrated by Dagster often power dashboards and other assets in BI tools. The lineage between these BI assets and the assets they depend on is crucial for understanding where dashboard data comes from, how data is being used, and for debugging issues when they arise.

These integrations represent assets in these BI tools in the Dagster asset graph. They also enable you to orchestrate updates to these assets, ensuring your BI tools reflect the most current data from upstream sources. This makes managing data refreshes simpler and more reliable, without relying on clunky cron jobs or manual updates.

To learn more, visit the documentation page for one of these integrations:

Airlift

We’ve introduced a preview of Airlift, a toolkit to accelerate, lower the cost of, and reduce the risk of migrating from Airflow to Dagster.

It facilitates a three-step, incremental process:

Peer: With a single line of code, observe Airflow DAGs and run history within Dagster.
Observe: Model the lineage of assets orchestrated by Airflow within Dagster and track their execution without modifying Airflow code.
Migrate: With minimal changes to Airflow code, invoke Dagster via an API from within Airflow. This is configurable per-task, enabling migration in any order while keeping Airflow DAG structure and execution history intact during the process.

We’ve seen early success with the approach, and are looking for further feedback and additional design partners.

Lean more in the Github Discussion.

UI

Information architecture changes

We’ve made several changes to the information architecture to make important entities easier to find:

Backfills have been moved from their own tab underneath the Overview page to entries within the table on the Runs page. This reflects the fact that backfills and runs are similar entities that share most properties. GitHub Discussion.
“Jobs” is now a page reachable from the top-level navigation pane. It replaces the Jobs tab within the Overview page.
“Automations” is now a page reachable from the top-level navigation pane. It replaces the schedule and sensor tabs within the Overview page.

Asset kinds

Assets have a more general kinds attribute that enables specifying labels that show up on asset nodes in the asset graph in the UI. This supersedes the compute_kind attribute.