Dagster Integration:
Using Dagster with Kubernetes

About this integration

The dagster-k8s integration library provides the PipesK8sClient resource, enabling you to launch Kubernetes pods and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Kubernetes pods while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement.

Installation

pip install dagster-k8s

Example

import dagster as dg
from dagster_k8s import PipesK8sClient


@dg.asset
def k8s_pipes_asset(
    context: dg.AssetExecutionContext, k8s_pipes_client: PipesK8sClient
):
    return k8s_pipes_client.run(
        context=context,
        image="pipes-example:v1",
    ).get_materialize_result()


defs = dg.Definitions(
    assets=[k8s_pipes_asset],
    resources={
        "k8s_pipes_client": PipesK8sClient(),
    },
)

Deploying to Kubernetes?

• Deploying to Dagster+: Use with a Dagster+ Hybrid deployment, the Kubernetes agent executes Dagster jobs on a Kubernetes cluster. Checkout the Dagster+ Kubernetes Agent guide for more information.
• Deploying to Open Source: Visit the Deploying Dagster to Kubernetes guide for more information.

About Kubernetes

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management. Google originally designed Kubernetes, but the Cloud Native Computing Foundation now maintains the project.