Using Airbyte with Dagster
About this integration
Using this integration, you can trigger Airbyte syncs and orchestrate your Airbyte connections from within Dagster, making it easy to chain an Airbyte sync with upstream or downstream steps in your workflow.
The guide focuses on how to work with Airbyte connections using Dagster's software-defined asset (SDA) framework.
pip install dagster-airbyte
from dagster import EnvVar from dagster_airbyte import AirbyteResource, load_assets_from_airbyte_instance import os # Connect to your OSS Airbyte instance airbyte_instance = AirbyteResource( host="localhost", port="8000", # If using basic auth, include username and password: username="airbyte", password=EnvVar("AIRBYTE_PASSWORD") ) # Load all assets from your Airbyte instance airbyte_assets = load_assets_from_airbyte_instance(airbyte_instance)
Airbyte is an open-source data integration engine that helps you consolidate your SaaS application and database data into your data warehouses, lakes and databases.