About this integration

The dagster-docker integration enables APIs for launching containers and launching steps as Docker containers.

The Deploying Dagster to Docker guide will walk you through a minimal skeleton Dockerfile for single container deployment as well as an advanced deployment guide for deploying more than one container. It also provides context for launching runs in containers and mounting your code as a volume.

Installation

pip install dagster-docker

Example

# Materialize your assets in local Docker containers
# Read the docs on Executors to learn more: https://docs.dagster.io/deployment/executors

from dagster_docker import docker_executor
from dagster import define_asset_job
import os

executor = docker_executor.configured({
  'image': 'my_repo.com/image_name:latest',
  'registry': {
    'url': 'my_repo.com',
    'username': 'my_user',
    'password': os.environ['DOCKER_REGISTRY_PASSWORD'],
  },
  'env_vars': ['DAGSTER_HOME'], # environment vars to pass from celery worker to docker
  'container_kwargs': { # keyword args to be passed to the container. example:
    'volumes': ['/home/user1/:/mnt/vol2', '/var/www:/mnt/vol1'],
  },
})

docker_enabled_job = define_asset_job("docker_enabled_job", executor_def=executor)

About Docker

Docker is a set of platform-as-a-service products that use OS-level virtualization to deliver software in packages called containers. The service has both free and premium tiers. The software that hosts the containers is called Docker Engine.