About this integration

This library contains utilities for running Dagster with Kubernetes. This includes a Python API allowing Dagit to launch runs as Kubernetes Jobs, as well as a Helm chart you can use as the basis for a Dagster deployment on a Kubernetes cluster.

Installation

pip install dagster-k8s

Example

from dagster import job
from dagster_k8s import k8s_job_op

first_op = k8s_job_op.configured(
    {
        "image": "busybox",
        "command": ["/bin/sh", "-c"],
        "args": ["echo HELLO"],
        "kubeconfig_file": "path/to/kubeconfig",
    },
    name="first_op",
)
second_op = k8s_job_op.configured(
    {
        "image": "busybox",
        "command": ["/bin/sh", "-c"],
        "args": ["echo GOODBYE"],
        "kubeconfig_file": "path/to/kubeconfig",
    },
    name="second_op",
)

@job
def full_job():
    second_op(first_op())

About Kubernetes

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management. Google originally designed Kubernetes, but the Cloud Native Computing Foundation now maintains the project.