Dagster Integration:
Using AWS Lambda with Dagster

About this integration

Using this integration, you can leverage AWS Lambda to execute external code as part of your Dagster pipelines. This is particularly useful for running serverless functions that can scale automatically and handle various workloads without the need for managing infrastructure. The PipesLambdaClient class allows you to invoke AWS Lambda functions and stream logs and structured metadata back to Dagster's UI and tools.

Installation

pip install dagster-aws

Examples

import boto3

from dagster import asset, AssetExecutionContext, Definitions
from dagster_aws.pipes import PipesLambdaClient


lambda_client = boto3.client("lambda", region_name="us-west-1")

lambda_pipes_client = PipesLambdaClient(client=lambda_client)


@asset
def lambda_pipes_asset(
    context: AssetExecutionContext, lambda_pipes_client: PipesLambdaClient
):
    return lambda_pipes_client.run(
        context=context,
        function_name="your_lambda_function_name",
        event={"key": "value"},
    ).get_materialize_result()


defs = Definitions(
    assets=[lambda_pipes_asset],
    resources={"lambda_pipes_client": lambda_pipes_client},
)

About AWS Lambda

AWS Lambda is a serverless compute service provided by Amazon Web Services (AWS). It allows you to run code without provisioning or managing servers. AWS Lambda automatically scales your application by running code in response to each trigger, such as changes to data in an Amazon S3 bucket or an update to a DynamoDB table. You can use AWS Lambda to extend other AWS services with custom logic, or create your own backend services that operate at AWS scale, performance, and security.