Dagster Integration:
Using Dagster with Docker

About this integration

The dagster-docker integration library provides the PipesDockerClient resource, enabling you to launch Docker containers and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Docker containers while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement.

Installation

pip install dagster-docker

Example

import dagster as dg
from dagster_docker import PipesDockerClient


@dg.asset
def docker_pipes_asset(
    context: dg.AssetExecutionContext, docker_pipes_client: PipesDockerClient
):
    docker_image = "python:3.9-slim"
    return docker_pipes_client.run(
        image=docker_image,
        command=[
            "python",
            "-m",
            "my_module",
        ],
        context=context,
    ).get_results()


defs = dg.Definitions(
    assets=[docker_pipes_asset],
    resources={
        "docker_pipes_client": PipesDockerClient(),
    },
)

Deploying to Docker?

• Deploying to Dagster+: Use with a Dagster+ Hybrid deployment, the Docker agent executes Dagster jobs on a Docker cluster. Checkout the Dagster+ Docker Agent guide for more information.
• Deploying to Open Source: Visit the Deploying Dagster to Docker guide for more information.

About Docker

Docker is a set of platform-as-a-service products that use OS-level virtualization to deliver software in packages called containers. The service has both free and premium tiers. The software that hosts the containers is called Docker Engine.