Dagster Integration:
Run Meltano jobs from Dagster

About this integration

The dagster-meltano library allows you to run Meltano using Dagster. Design and configure ingestion jobs using the popular Singer.io specification.

Note that this integration can also be managed from the Meltano platform using meltano add utility dagster and configured using meltano config dagster set --interactive.

Installation

pip install dagster-meltano

Example

An example of running an abitrary Meltano run command:

from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_run_op

@job(resource_defs={"meltano": meltano_resource})
def meltano_run_job():
    tap_done = meltano_run_op("tap-1 target-1")()
    meltano_run_op("tap-2 target-2")(tap_done)

@repository()
def repository():
    return [meltano_run_job]

About Meltano

Meltano provides data engineers with a set ot tools for easily creating and managing pipelines as code by providing a wide array of composable connectors. Meltano's 'CLI for ELT+' lets you test your changes before they go live.

About Quantile

Quantile is an expert data consultancy based in the Netherlands. You can find out more at quantile.nl.