Dagster + Anthropic

About dagster-anthropic

The dagster-anthropic library allows you to easily interact with the Anthropic REST API using the Anthropic Python API to build AI steps into your Dagster pipelines. You can also log Anthropic API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption.

When paired with Dagster assets, the resource automatically logs Anthropic usage metadata in asset metadata.

Installation

pip install dagster dagster-anthropic

Example

from dagster_anthropic import AnthropicResource
import dagster as dg

@dg.asset(compute_kind="anthropic")
def anthropic_asset(context: dg.AssetExecutionContext, anthropic: AnthropicResource):
    with anthropic.get_client(context) as client:
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=1024,
            messages=[{"role": "user", "content": "Say this is a test"}],
        )
    return response

defs = dg.Definitions(
    assets=[anthropic_asset],
    resources={
        "anthropic": AnthropicResource(api_key=dg.EnvVar("ANTHROPIC_API_KEY")),
    },
)

About Anthropic

Anthropic is an AI research company focused on developing safe and ethical AI systems. Their Claude models are designed to be helpful, harmless, and honest, with a strong emphasis on safety and alignment with human values.

‍