Join us for a Dagster Deep Dive with Artefact - Dec 19

Back to integrations
OpenAI

Dagster Integration:
OpenAI

Integrate OpenAI calls into your Dagster pipelines, without breaking the bank.
Docs overviewAPI reference

About dagster-openai

The dagster-openai library allows you to easily interact with the OpenAI REST API via the OpenAI Python API to build AI steps into your Dagster pipelines. You can also log OpenAI API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption.

When paired with Dagster's Software-defined Assets, the resource automatically logs OpenAI usage metadata in asset metadata.

Installation

pip install dagster dagster-openai

Example

from dagster_openai import OpenAIResource

from dagster import (
    AssetExecutionContext,
    Definitions,
    EnvVar,
    asset,
    define_asset_job,
)


@asset(compute_kind="OpenAI")
def openai_asset(context: AssetExecutionContext, openai: OpenAIResource):
    with openai.get_client(context) as client:
        client.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[{"role": "user", "content": "Say this is a test."}],
        )


openai_asset_job = define_asset_job(name="openai_asset_job", selection="openai_asset")

defs = Definitions(
    assets=[openai_asset],
    jobs=[openai_asset_job],
    resources={
        "openai": OpenAIResource(api_key=EnvVar("OPENAI_API_KEY")),
    },
)

About OpenAI

OpenAI is a U.S. based artificial intelligence (AI) research organization with the goal of developing "safe and beneficial" artificial general intelligence, which it defines as "highly autonomous systems that outperform humans at most economically valuable work".