About this integration

With the dagster-celery integration you can execute and monitor Dagster jobs as Celery tasks.

Installation

pip install dagster-celery

Example

# Materialize your assets with Celery
# Read the docs on Executors to learn more: https://docs.dagster.io/deployment/executors

from dagster_celery import celery_executor
from dagster import define_asset_job

executor = celery_executor.configured({
  'broker': 'pyamqp://guest@localhost//', # The URL of the Celery broker
  'backend': 'rpc://', # The URL of the Celery results backend
  'include': ['my_module'], # Modules every worker should import
})

celery_enabled_job = define_asset_job("celery_enabled_job", executor_def=executor)

About Celery

Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling. The Celery Executor makes it easier for developers to scale up applications by distributing the tasks on multiple machines.