Databricks with Dagster Pipes

You can run code in Databricks while streaming logs and structured metadata back to Dagster in real time with Dagster Pipes. Choose a guide based on your cluster type, or on whether the job already exists in Databricks:

Guide	When to use
Serverless compute	Unity Catalog Volumes, serverless notebooks
DBFS, classic clusters	Python scripts on DBFS with classic (non-serverless) clusters
Existing jobs (workflows)	Jobs already defined in Databricks, triggered by job ID