Using dlt with Dagster

About this integration

This integration allows you to use dlt to easily ingest and replicate data between systems through Dagster.

Installation

pip install dagster-embedded-elt

Example

import dagster as dg
from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
from dlt import pipeline
from dlt_sources.github import github_reactions


@dlt_assets(
    dlt_source=github_reactions("dagster-io", "dagster"),
    dlt_pipeline=pipeline(
        pipeline_name="github_issues",
        dataset_name="github",
        destination="snowflake",
    ),
    name="github",
    group_name="github",
)
def github_issues_to_snowflake_assets(
    context: dg.AssetExecutionContext, dlt: DagsterDltResource
):
    yield from dlt.run(context=context)


defs = dg.Definitions(
    assets=[
        github_issues_to_snowflake_assets,
    ],
    resources={
        "dlt": DagsterDltResource(),
    },
)

About dlt

Data Load Tool (dlt) is an open-source library for creating efficient data pipelines. It offers features like secret management, data structure conversion, incremental updates, and pre-built sources and destinations, simplifying the process of loading messy data into well-structured datasets.