About this integration

Dagster helps you use Azure Storage Accounts as part of your data pipeline. Azure Data Lake Storage Gen 2 (ADLS2) is our primary focus but we also provide utilities for Azure Blob Storage.

Installation

pip install dagster-azure

Examples

# Store your software-defined assets in ADLS2

from dagster import Definitions, asset
from dagster_azure.adls2 import ADLS2Resource, ADLS2PickleIOManager
import pandas as pd

@asset
def asset1():
    return pd.DataFrame()

@asset
def asset2(asset1):
    return df[:5]

defs = Defintions(
    assets=[asset1, asset2],
    resources={
        "io_manager": ADLS2PickleIOManager(
            adls2_file_system="my-cool-fs",
            adls2_prefix="my-cool-prefix",
            adls2=ADLS2Resource()
        )
    },
)

About Azure

Microsoft Azure is a cloud computing platform operated by Microsoft for application management via Microsoft-managed data centers.