Classifying a Million Snowflake Columns in 9 Days, Solo, with Dagster
data governance. I built a tiered AI classification system, human review workflow, and the Dagster orchestration that ties it all together in production in nine days.
Oops! Something went wrong while submitting the form.
November 30, 2022
Getting Stuff Done: a Guide to Productive Software Engineering
To be a more productive software engineer you need to master changes, how these affect the program and others on the team.
Alex Langenfeld
Engineering
November 11, 2022
Pushing REST-API data to Google Sheets with Dagster
A total beginners tutorial in which we store REST API data in Google Sheets and learn some key abstractions.
Fraser Marlow
Engineering
November 7, 2022
Adding Types to a Large Python Codebase
What we learned when we introduced dynamically typed code to a large Python codebase, bringing Dagster's public API to 100% type coverage.
Sean Mackesey
Engineering
October 31, 2022
Orchestrating Machine Learning Pipelines with Dagster
How to use Dagster’s open source data orchestrator to build machine learning pipelines and train ML models.
Sandy Ryza
Engineering
October 25, 2022
Build a poor man’s data lake from scratch with DuckDB
DuckDB is so hot right now. Learn how to build a data lake from dbt using DuckDB for SQL transformations, along with Python, Dagster, and Parquet files.
Pete Hunt
Engineering
October 19, 2022
The Unreasonable Effectiveness of Data Pipeline Smoke Tests
Data practitioners waste time writing unit tests to catch bugs they could have caught with smoke tests.
Sandy Ryza
Engineering
October 17, 2022
Web Workers are not the Answer
A tale of overstretched logs, counterintuitive web worker behavior, and ultimately a troublesome cursor issue.
Jordan Sanders
Engineering
October 4, 2022
Postgres: a Better Message Queue than Kafka?
When lots of event logs must be stored and indexed, Kafka is the obvious choice. Naturally, our queue runs on Postgres.
Pete Hunt
Engineering
August 17, 2022
Spend Less Time Debugging with Dagster
It’s not uncommon for a data engineer to devote 80% of their day to debugging. Dagster radically improves on this.
Sandy Ryza
Engineering
August 3, 2022
The Open Core Business Model
The relationship between Dagster, the open-source project, and Dagster Cloud, our hosted SaaS platform.
