Airflow

About this integration

Airlift is a migration toolkit that makes it easy to transition from Airflow to Dagster with minimal disruption to your existing workflows. Rather than requiring a complete rewrite, Airlift enables a gradual, task-by-task migration that can be completed in any order with minimal coordination between teams.

With Airlift, you can:

• Observe multiple Airflow instances from a centralized Dagster control plane
• Migrate individual tasks or entire DAGs incrementally
• Federate execution across multiple Airflow deployments
• Monitor both Airflow and Dagster assets in a unified interface

The toolkit is designed to work with your existing Airflow infrastructure, including AWS MWAA, Google Cloud Composer, and Astronomer deployments.

Installation

pip install 'dagster-airlift[core]' dagster-webserver dagster

Key Features

Incremental Migration

Migrate your Airflow DAGs to Dagster assets one task at a time, allowing teams to move at their own pace without breaking existing workflows.

Multi-Instance Observability

Connect multiple Airflow instances to a single Dagster deployment for centralized monitoring and orchestration across your entire data platform.

Asset-Centric Approach

Transform Airflow tasks into Dagster software-defined assets, providing better lineage tracking and dependency management.

Minimal Code Changes

Existing Airflow code requires minimal modifications during the migration process, reducing risk and development overhead.

Getting Started

Step 1: Peer Your Airflow Instance

Connect your Airflow instance to Dagster to begin observing DAG executions:

from dagster_airlift.core import (
    AirflowBasicAuthBackend,
    AirflowInstance,
    build_defs_from_airflow_instance,
)

# Define your Airflow instance
airflow_instance = AirflowInstance(
    auth_backend=AirflowBasicAuthBackend(
        webserver_url="http://localhost:8080",
        username="admin",
        password="admin",
    ),
    name="my_airflow_instance",
)

# Create Dagster definitions
defs = build_defs_from_airflow_instance(
    airflow_instance=airflow_instance
)

Step 2: Observe Assets

Map your Airflow tasks to Dagster assets to track data lineage:

from dagster import AssetSpec
from dagster_airlift.core import assets_with_task_mappings

# Define asset specs for your data
customers_spec = AssetSpec(key="customers")
orders_spec = AssetSpec(key="orders") 

# Map assets to Airflow tasks
mapped_assets = assets_with_task_mappings(
    dag_id="etl_pipeline",
    task_mappings={
        "extract_customers": [customers_spec],
        "extract_orders": [orders_spec],
    }
)

Step 3: Migrate Tasks

Gradually migrate task execution from Airflow to Dagster:

# In your Airflow DAG file
from dagster_airlift.in_airflow import proxying_to_dagster
from dagster_airlift.in_airflow.proxied_state import load_proxied_state_from_yaml

# Enable proxying for specific tasks
if PROXYING_ENABLED:
    proxying_to_dagster(
        global_vars=globals(),
        proxied_state=load_proxied_state_from_yaml(
            Path(__file__).parent / "proxied_state.yaml"
        ),
    )

Supported Airflow Versions

Airlift supports Airflow 2.0+ and requires access to Airflow's REST API:

• Airflow 2.0+: Full support with stable REST API
• AWS MWAA: Supported (Airflow 2.4.3 or later required)
• Google Cloud Composer: Full support
• Astronomer: Full support
• Self-hosted Airflow: Full support with proper API access

About Airlift

Airlift represents a paradigm shift in data platform migration, moving away from "big bang" rewrites toward incremental, low-risk transitions. By leveraging Airflow's REST API and Dagster's software-defined assets, Airlift provides a bridge between traditional workflow orchestration and modern data platform architecture.

The toolkit was designed based on real-world migration experiences, focusing on practical challenges like organizational coordination, system reliability, and business continuity during platform transitions.

‍