# Airflow

> Accelerate the migration of Airflow DAGs to Dagster assets with opinionated tooling.

Source: https://dagster.io/integrations/airflow
Category: Other

## About this integration

Airlift is a migration toolkit that makes it easy to transition from Airflow to Dagster with minimal disruption to your existing workflows. Rather than requiring a complete rewrite, Airlift enables a gradual, task-by-task migration that can be completed in any order with minimal coordination between teams.

With Airlift, you can:

- **Observe** multiple Airflow instances from a centralized Dagster control plane
- **Migrate** individual tasks or entire DAGs incrementally
- **Federate** execution across multiple Airflow deployments
- **Monitor** both Airflow and Dagster assets in a unified interface

The toolkit is designed to work with your existing Airflow infrastructure, including AWS MWAA, Google Cloud Composer, and Astronomer deployments.

## Installation

```bash
pip install 'dagster-airlift[core]' dagster-webserver dagster
```

## Key Features

### Incremental Migration

Migrate your Airflow DAGs to Dagster assets one task at a time, allowing teams to move at their own pace without breaking existing workflows.

### Multi-Instance Observability

Connect multiple Airflow instances to a single Dagster deployment for centralized monitoring and orchestration across your entire data platform.

### Asset-Centric Approach

Transform Airflow tasks into Dagster software-defined assets, providing better lineage tracking and dependency management.

### Minimal Code Changes

Existing Airflow code requires minimal modifications during the migration process, reducing risk and development overhead.

## Getting Started

### Step 1: Peer Your Airflow Instance

Connect your Airflow instance to Dagster to begin observing DAG executions:

```python
from dagster_airlift.core import (
    AirflowBasicAuthBackend,
    AirflowInstance,
    build_defs_from_airflow_instance,
)

# Define your Airflow instance
airflow_instance = AirflowInstance(
    auth_backend=AirflowBasicAuthBackend(
        webserver_url="http://localhost:8080",
        username="admin",
        password="admin",
    ),
    name="my_airflow_instance",
)

# Create Dagster definitions
defs = build_defs_from_airflow_instance(
    airflow_instance=airflow_instance
)
```

### Step 2: Observe Assets

Map your Airflow tasks to Dagster assets to track data lineage:

```python
from dagster import AssetSpec
from dagster_airlift.core import assets_with_task_mappings

# Define asset specs for your data
customers_spec = AssetSpec(key="customers")
orders_spec = AssetSpec(key="orders") 

# Map assets to Airflow tasks
mapped_assets = assets_with_task_mappings(
    dag_id="etl_pipeline",
    task_mappings={
        "extract_customers": [customers_spec],
        "extract_orders": [orders_spec],
    }
)
```

## Step 3: Migrate Tasks

Gradually migrate task execution from Airflow to Dagster:

```python
# In your Airflow DAG file
from dagster_airlift.in_airflow import proxying_to_dagster
from dagster_airlift.in_airflow.proxied_state import load_proxied_state_from_yaml

# Enable proxying for specific tasks
if PROXYING_ENABLED:
    proxying_to_dagster(
        global_vars=globals(),
        proxied_state=load_proxied_state_from_yaml(
            Path(__file__).parent / "proxied_state.yaml"
        ),
    )
```

## Supported Airflow Versions

Airlift supports Airflow 2.0+ and requires access to Airflow's REST API:

- **Airflow 2.0+**: Full support with stable REST API
- **AWS MWAA**: Supported (Airflow 2.4.3 or later required)
- **Google Cloud Composer**: Full support
- **Astronomer**: Full support
- **Self-hosted Airflow**: Full support with proper API access

## About Airlift

Airlift represents a paradigm shift in data platform migration, moving away from "big bang" rewrites toward incremental, low-risk transitions. By leveraging Airflow's REST API and Dagster's software-defined assets, Airlift provides a bridge between traditional workflow orchestration and modern data platform architecture.

The toolkit was designed based on real-world migration experiences, focusing on practical challenges like organizational coordination, system reliability, and business continuity during platform transitions.