Dagster + Clickhouse

About dagster-clickhouse

The dagster-clickhouse library provides an integration with ClickHouse using the native protocol (clickhouse-driver). Use a resource for ad hoc SQL, I/O managers for Pandas or Polars DataFrames, and components to wire templated SQL into Dagster projects.

Installation

pip install dagster dagster-clickhouse

Example

import pandas as pd
from dagster_clickhouse_pandas import (
    ClickhousePandasIOManager,
)

import dagster as dg


@dg.asset
def iris_dataset() -> (
    pd.DataFrame
):  # asset name is the table name; I/O manager `schema` is the ClickHouse database
    return pd.read_csv(
        "https://docs.dagster.io/assets/iris.csv",
        names=[
            "sepal_length_cm",
            "sepal_width_cm",
            "petal_length_cm",
            "petal_width_cm",
            "species",
        ],
    )


defs = dg.Definitions(
    assets=[iris_dataset],
    resources={
        "io_manager": ClickhousePandasIOManager(
            host="localhost",
            port=9000,
            database="default",
            schema="iris",
        )
    },
)

About Clickhouse

ClickHouse is a column-oriented OLAP database designed for analytical queries over large datasets.