194 Posts
Sep 23, 2024
Dagster vs. Airflow
Get the tale of the tape between the two orchestration giants and see why Dagster stands tall as the superior choice.
- Name
- TéJaun RiChard
- Handle
- @tejaun
- Name
- Sandy Ryza
- Handle
- @s_ryz
Oct 3, 2024
10 Reasons Why No-Code Solutions Almost Always Fail
No-code solutions sound easy – until they aren’t. Here’s why they often fail and what you can do about it for your data ...
- Name
- TéJaun RiChard
- Handle
- @tejaun
Sep 30, 2024
5 Best Practices AI Engineers Should Learn From Data Engineering
AI engineering is data engineering. Here are 5 best practices the former should adopt from the latter to succeed.
- Name
- TéJaun RiChard
- Handle
- @tejaun
Sep 27, 2024
Dagster Deep Dive Recap: Orchestrating Flexible Compute for ML with Dagster and Modal
Learn how to use Dagster and Modal to automate and streamline your machine learning model training and data processing.
- Name
- TéJaun RiChard
- Handle
- @tejaun
Sep 26, 2024
The Rise of the Data Platform Engineer
How the next step in the evolution of the Data Engineering role requires a platform approach.
- Name
- Pedram Navid
- Handle
- @pdrmnvd
Sep 16, 2024
Sakila Co.: An End-to-End Open-Source Analytics Starter Project
Jumpstart your analytics work with some of today’s best open-source technologies.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Sep 12, 2024
Data Visibility -- A Primer
The unseen data is often the deadliest. Here’s how to shine a light on it in your business.
- Name
- TéJaun RiChard
- Handle
- @tejaun
Sep 6, 2024
Dagster Deep Dive Recap: Building a True Data Platform
Move past the MDS and build a data platform for observability, cost-efficiency, and top-tier orchestrating.
- Name
- TéJaun RiChard
- Handle
- @tejaun
Sep 4, 2024
Case Study: Mejuri - Building an eCommerce Data Platform
Mejuri’s nimble business model requires a rock-solid data platform to support the company’s rapid growth.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Aug 30, 2024
Dagster Deep Dive Recap: Evolution of the Data Platform
Dagster and SDF show how the power of two can connect local development and production orchestration.
- Name
- TéJaun RiChard
- Handle
- @tejaun
Aug 15, 2024
Case Study: The Lean and Efficient One-Person Data Team of Erewhon
How a solo data team delivered a custom system to accelerate data transformation.
- Name
- Colton Padden
- Handle
- @colton
Aug 14, 2024
Combining Dagster and SDF: The Post-Modern Data Stack for End-to-End Data Platforms
Dagster orchestration meets SDF transformation to improve developer experience with transparent, efficient, pipelines.
- Name
- TéJaun RiChard
- Handle
- @tejaun
Aug 8, 2024
Dagster 1.8: Call Me Maybe
Ecosystem and integration improvements, data catalog improvements, new asset checks, new declarative automation, and ...
- Name
- TéJaun RiChard
- Handle
- @tejaun
Aug 7, 2024
Dagster Deep Dive Recap: Building Reliable Data Platforms
Explore the importance of data quality and learn strategies for integrating quality checks using Dagster.
- Name
- TéJaun RiChard
- Handle
- @tejaun
- Name
- Colton Padden
- Handle
- @colton
Jul 29, 2024
Case Study: Artemis - Powering the Crypto Markets
Artemis built a data platform around Dagster+ to bring consolidated reporting to the $2.5T Cryptocurrency markets.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Jul 24, 2024
Case Study: How Petal Incrementally Adopted a Data Orchestrator
How Petal’s incremental adoption of Dagster let this FinTech firm build out its data platform at its own speed.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Jul 18, 2024
A Look Inside the Dagster Labs Culture
Operations Lead Eunice Ho dives into the Dagster Labs culture and why it makes for an ideal work environment.
- Name
- Eunice Ho
- Handle
- @eunice
Jul 8, 2024
Enabling Data Quality with Dagster and Great Expectations
Use Dagster and GX to improve data pipeline reliability without writing custom logic for data testing.
- Name
- Muhammad Jarir Kanji
- Handle
- @muhammad
Jul 5, 2024
Case Study: A Start-up’s Rite of Passage - Establishing the Data Platform
Zippi successfully navigated a common growth milestone, future-proofing data operations on Dagster.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Jun 21, 2024
Podcast: Value Driven Data Science - The Impact of Data Science on Data Orchestration
Sandy Ryza on the impact of data scientists on the creation of the next generation of data orchestration tools.
- Name
- Sandy Ryza
- Handle
- @s_ryz
Jun 10, 2024
The Rise of Medium Code
Why the reports of software’s demise are greatly exaggerated.
- Name
- Nick Schrock
- Handle
- @schrockn
Jun 7, 2024
Running Singer on Dagster
Singer Taps and Targets are popular data movement tools. Here is how (and why) you run them in Dagster.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Jun 5, 2024
ELT Options in Dagster
Why running data ingestion jobs straight from the orchestrator is often a preferred approach.
- Name
- TéJaun RiChard
- Handle
- @tejaun
- Name
- Fraser Marlow
- Handle
- @frasermarlow
May 28, 2024
Dagster’s Code Location Architecture
A structure for a reliable, maintainable data platform design.
- Name
- Pete Hunt
- Handle
- @floydophone
May 17, 2024
What is Dagster: A Guide to the Data Orchestrator
Get to know the tool that sets the standard for modern data orchestration.
- Name
- Pete Hunt
- Handle
- @floydophone
May 8, 2024
Building Cost Effective AI Pipelines with OpenAI, LangChain, and Dagster
Leverage the power of LLMs while keeping the costs in check using the Dagster OpenAI integration.
- Name
- Maxime Armstrong
- Handle
- @maxime
- Name
- Yuhan Luo
- Handle
- @yuhan
Apr 30, 2024
Unlocking Flexible Pipelines: Customizing the Asset Decorator
Use Asset Factories within Dagster to streamline data asset creation, promote code reusability, and maintain data ...
- Name
- Daniel Gafni
- Handle
- @danielgafni
Apr 17, 2024
See Both the Forest and the Trees with Dagster+ Insights
How Dagster+ Insights helps you control costs and elevate your data platform’s observability.
- Name
- Christian Minich
- Handle
- @christianminich
Apr 17, 2024
Ensuring Reliable Data with Dagster+
Dagster+ helps you monitor the freshness, quality, and schema of your data.
- Name
- Sandy Ryza
- Handle
- @s_ryz
Apr 17, 2024
Dagster+ Catalog: A New Built-in Asset Library for All Practitioners
Give your data teams a powerful new system of record without the overhead of maintaining a third-party catalog.
- Name
- Jarred Colli
- Handle
- @jarred
Apr 17, 2024
Change Tracking Branch Deployments in Dagster+
Dagster+ further enhances identification and collaboration around changes to your data pipelines.
- Name
- Jamie DeMaria
- Handle
Apr 11, 2024
Use Dagster and SkyPilot to Orchestrate Cost-Effective AI Training Jobs
Explore the efficient orchestration of AI training jobs with Dagster and SkyPilot.
- Name
- Muhammad Jarir Kanji
- Handle
- @muhammad
Apr 10, 2024
The Data Engineering Impedance Mismatch
A case for asset-oriented over workflow-oriented in data orchestration.
- Name
- Pete Hunt
- Handle
- @floydophone
Apr 8, 2024
Announcing Dagster 1.7: Love Plus One
A major set of updates to Dagster Core ahead of our Dagster+ launch.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Apr 5, 2024
Expanding the Dagster Embedded ELT Ecosystem with dltHub for Data Ingestion
We now have an officially supported dlt integration.
- Name
- Colton Padden
- Handle
- @colton
Apr 3, 2024
Sling Out Your ETL Provider with Embedded ELT
How we saved $40k and gained better control over our ingestion steps.
- Name
- Nick Roach
- Handle
Mar 26, 2024
Exploring The Data Engineering Lifecycle
Learn the fundamentals of a healthy data engineering lifecycle to optimize pipeline and asset production.
- Name
- Sandy Ryza
- Handle
- @s_ryz
Mar 22, 2024
How Dagster Cloud Supports BCBS 239 Compliance
BCBS 239 establishes standards for banking risk management worldwide. Dagster helps data engineers meet these demanding ...
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Mar 11, 2024
New Dagster Integration: Include OpenAI Calls Into Your Data Pipelines
The new dagster-openai integration lets you tap into the power of LLMs in a cost-efficient way.
- Name
- Yuhan Luo
- Handle
- @yuhan
- Name
- Maxime Armstrong
- Handle
- @maxime
Mar 10, 2024
Podcast: Tech Talks Daily - Data, Decisions, and Dagster
Nick Schrock shares his blueprint for engineering excellence on the Tech Talks Daily Podcast.
Mar 6, 2024
Dagster University Presents: Dagster & dbt™
Learn how to combine your dbt™ knowledge with Dagster’s asset-focused approach for an enhanced data platform experience.
- Name
- Erin Cochran
- Handle
Mar 2, 2024
How to Make Data a Team Sport
Enabling internal access and collaboration around data in organizations is vital to tackling data complexity.
- Name
- Colton Padden
- Handle
- @colton
- Name
- TéJaun RiChard
- Handle
- @tejaun
Feb 27, 2024
Breaking Packages in Python
An exposé of the nooks and crannies of Python’s modules and packages.
- Name
- Pedram Navid
- Handle
- @pdrmnvd
Feb 23, 2024
Balancing the Data Scales: Centralization vs. Decentralization
Learn how organizations can harness the strengths of both approaches to optimize their data operations.
- Name
- TéJaun RiChard
- Handle
- @tejaun
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Feb 20, 2024
Case Study: BenchSci - A Leap Forward with Dagster
Learn about how BenchSci uses Dagster in their journey to expedite drug development.
- Name
- TéJaun RiChard
- Handle
- @tejaun
Feb 17, 2024
Podcast: A Geek Leader - Interview with Nick Schrock
John Rouda interviewed Nick Schrock, Founder of Dagster Labs, on open-source, ML, and the future of Dagster.
Feb 15, 2024
Addressing Big Complexity Through Strategic Orchestration
For organizations looking to thrive in the era of Big Complexity, it’s time to reassess the role of orchestration in ...
- Name
- TéJaun RiChard
- Handle
- @tejaun
Feb 14, 2024
Podcast: Open Source Underdogs - Scaling Data Pipelines
Nick joins the Open Source Underdogs podcast for a conversation on how Dagster Labs is evolving.
- Name
- Nick Schrock
- Handle
- @schrockn
Feb 8, 2024
Standardize Pipelines with Domain-Specific Languages
By implementing DSLs, data teams can open their data platform to many more users without compromising on standards.
- Name
- Elliot Gunn
- Handle
- @elliot
- Name
- Tim Castillo
- Handle
- @tim
Feb 7, 2024
Podcast: Partially Redacted - Learning and Sharing in Public
Pedram Navid of Dagster Labs discusses the culture of learning and sharing in Data Engineering.
- Name
- Pedram Navid
- Handle
- @pdrmnvd
Feb 6, 2024
Podcast: Facebook Eng Culture & Modern Data Stack Consolidation
On open source software, data, and understanding Facebook’s high performance culture.
- Name
- Nick Schrock
- Handle
- @schrockn
Feb 5, 2024
Thinking in Assets When Building Data Pipelines
How to develop data pipelines using Software-defined Assets.
- Name
- Tim Castillo
- Handle
- @tim
- Name
- Sandy Ryza
- Handle
- @s_ryz
Jan 29, 2024
What Dagster Believes About Data Platforms
The beliefs that organizations adopt about the way their data platforms should function influence their outcomes. Here ...
- Name
- Sandy Ryza
- Handle
- @s_ryz
Jan 26, 2024
Podcast: Data Driven - The Role of AI and LLMs in Data
Pedram Navid fo Dagster Labs joins the Data Driven podcast to discuss the role of AI and LLMs in data.
- Name
- Pedram Navid
- Handle
- @pdrmnvd
Jan 26, 2024
Podcast: Data Driven - Cutting Through the Noise of Data Products
Pedram Navid of Dagster Labs talks about how data teams can strategically enable self-service to speed up business ...
- Name
- Pedram Navid
- Handle
- @pdrmnvd
Jan 12, 2024
Announcing Dagster 1.6: Back to Black
Major UI enhancements, Dagster Pipes upgrades and of course, dark mode :-)
- Name
- Sandy Ryza
- Handle
- @s_ryz
Jan 10, 2024
Retain.ai joins Dagster Labs
We’re excited and humbled to bring the Retain.ai organization into our fold to help build out Dagster’s data ...
- Name
- Pete Hunt
- Handle
- @floydophone
Jan 3, 2024
Podcast: Machine Learning Pipelines Are Still Data Pipelines
Sandy Ryza, Lead Engineer at Dagster Labs, talks data engineering for machine learning efforts.
- Name
- Sandy Ryza
- Handle
- @s_ryz
Dec 21, 2023
Podcast: Alter Everything - The Present & Future of Data Engineering
Nick Schrock joins the Alteryx podcast about data science and analytics culture.
- Name
- Nick Schrock
- Handle
- @schrockn
Dec 4, 2023
How Dagster Labs runs Dagster: Open-Sourcing our Own Pipelines
A technical deep dive into the patterns and implementations of the Dagster Open Platform using our open-sourced code ...
- Name
- Tim Castillo
- Handle
- @tim
Nov 29, 2023
Scaling Dagster’s DAG Visualization to Handle Tens of Thousands of Assets
How the Dagster frontend team rapidly scaled Dagster’s DAG visualization for enterprise-sized data asset graphs.
- Name
- Marco Salazar
- Handle
- @BkOptimism
Nov 28, 2023
Case Study: Abstracting Pipelines for Analysts with a YAML DSL
How SimpliSafe’s small engineering team uses YAML DSL within Dagster’s powerful data platform to support analysts and ...
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Nov 20, 2023
High-performance Python for Data Engineering
Learn how to optimize your Python data pipeline code to run faster with our high-performance Python guide for data ...
- Name
- Elliot Gunn
- Handle
- @elliot
Nov 14, 2023
Podcast: That Tech Pod - Pete Hunt's Engineering Journey
The Journey from Engineer to CEO and Lessons Learned Along the Way
- Name
- Pete Hunt
- Handle
- @floydophone
Nov 8, 2023
Orchestrate Unstructured Data Pipelines with Dagster and dlt
Load messy data sources into well-structured tables or datasets, through automatic schema inference and evolution.
- Name
- Zaeem Athar
- Handle
- @zaeem
Oct 31, 2023
Podcast: The Craft Of Open Source - a Flagsmith podcast
Pete Hunt discusses data orchestration, Dagster, and our onward journey.
- Name
- Pete Hunt
- Handle
- @floydophone
Oct 31, 2023
Podcast: Data Unlocked - How to Work Effectively With Your Data Teams
Nick Schrock on the relationship between data engineering and go-to-market.
- Name
- Nick Schrock
- Handle
- @schrockn
Oct 20, 2023
CI/CD and Data Pipeline Automation (with Git)
Learn how to automate data pipelines and deployments by integrating Git and CI/CD in our Python for data engineering ...
- Name
- Elliot Gunn
- Handle
- @elliot
Oct 19, 2023
Podcast: The Tech Trek Podcast - Open source data orchestration
Pete Hunt shares insights on the challenges in the data orchestration market, and why Dagster is open-source.
- Name
- Pete Hunt
- Handle
- @floydophone
Oct 13, 2023
Introducing Dagster Pipes
A new protocol and toolkit for integrating and launching compute into remote execution environments from Dagster.
- Name
- Nick Schrock
- Handle
- @schrockn
Oct 13, 2023
Introducing Dagster External Assets
Use Dagster’s External Assets feature for data observability, lineage, data quality, and cataloging while bringing your ...
- Name
- Nick Schrock
- Handle
- @schrockn
Oct 12, 2023
Stop Reinventing Orchestration: Embedded ELT in the Orchestrator
Solve data ingestion issues with Dagster's Embedded ELT feature, a lightweight embedded library.
- Name
- Pedram Navid
- Handle
- @pdrmnvd
Oct 11, 2023
Improving the Dagster learning curve
Learn Dagster essentials and build asset-based data pipelines with Dagster University, our new self-guided course for ...
- Name
- Erin Cochran
- Handle
Oct 10, 2023
Improving visibility into data operations with Dagster Insights
Gain operational observability on your data pipelines and bring cloud costs back under control with the Dagster ...
- Name
- Jarred Colli
- Handle
- @jarred
Oct 9, 2023
Introducing Dagster Asset Checks
Deliver high-quality data with Dagster Asset Checks, the ability to embed data quality checks into your data pipeline.
- Name
- Sandy Ryza
- Handle
- @s_ryz
- Name
- Johann Miller
- Handle
- @johann
Oct 4, 2023
Podcast: The Orchestration Layer as the Data Platform Control Plane
Nick Schrock, founder and CTO of Dagster Labs, discusses the data platform control plane on The Data Stack Show.
- Name
- Nick Schrock
- Handle
- @schrockn
Oct 2, 2023
Announcing Dagster 1.5: How Will I Know?
Ahead of Launch Week, we are proud to be rolling out some exciting new capabilities.
- Name
- Yuhan Luo
- Handle
- @yuhan
Sep 29, 2023
Write-Audit-Publish in data pipelines
We look at the write-audit-publish software design pattern used in ETL to ensure quality and reliability in data ...
- Name
- Elliot Gunn
- Handle
- @elliot
Sep 28, 2023
Escaping the Modern Data Trap
Launch Week kicks off October 9th with new functionality being shared each day. Our theme: Escaping the Modern Data ...
- Name
- Pete Hunt
- Handle
- @floydophone
- Name
- Nick Schrock
- Handle
- @schrockn
Sep 21, 2023
Podcast: Open Source Startup - Bringing Great Developer Experience to Data Teams
Nick Schrock on how Dagster is bringing software engineering principles to the data space, and what a great developer ...
- Name
- Nick Schrock
- Handle
- @schrockn
Sep 20, 2023
Pedram Navid: Why I Joined Dagster Labs
It is not every day you get to join a company working on building a product purpose-built for you.
- Name
- Pedram Navid
- Handle
- @pdrmnvd
Sep 14, 2023
A Dagster-Powered Spam Filter
Using Dagster, you can maintain data trust and protect the integrity of any user-generated service with this powerful ...
- Name
- James Timmins
- Handle
- @jamestimmins
Sep 13, 2023
Podcast: Code Story - The Origin Story of Dagster
Pete Hunt joins Noah Labhart - startup founder & CTO - to discuss the origin story of Dagster.
- Name
- Pete Hunt
- Handle
- @floydophone
Sep 10, 2023
Podcast: Data Orchestration in an Increasingly Complex Data Ecosystem
Nick Schrock shares his perspective on the state of data orchestration technology and its application to help inform ...
- Name
- Nick Schrock
- Handle
- @schrockn
Sep 4, 2023
Factory Patterns in Python
We explore design patterns — reusable solutions to common problems in software design — as used in data engineering, ...
- Name
- Elliot Gunn
- Handle
- @elliot
Aug 29, 2023
Migrating off dbt Cloud™
Looking for an alternative tool to orchestrate your dbt projects? Here’s a step-by-step guide to migrating from dbt ...
- Name
- Tim Castillo
- Handle
- @tims_tangents
- Name
- Claire Lin
- Handle
Aug 28, 2023
Podcast: The Breakthrough Hiring Show with Pete Hunt
Pete and host James Mackey discuss strategic hiring for startups and the dangers of getting too big too fast.
- Name
- Pete Hunt
- Handle
- @floydophone
Aug 28, 2023
ML pipelines for fine-tuning LLMs
LLM fine-tuning best practices for creating a clean production ML pipeline, streamlining model training, and ...
- Name
- Odette Harary
- Handle
- @odette
Aug 24, 2023
Podcast: The Happy Engineer Podcast - Engineering Hard Choices
Pete Hunt shares insights on building and leading a data engineering team and making hard engineering calls.
- Name
- Pete Hunt
- Handle
- @floydophone
Aug 24, 2023
Podcast: Adventures in DevOps - Testing and Development in the Data Domain
The Adventures in DevOps podcast chats with Pete Hunt about testing and development in the data domain
- Name
- Pete Hunt
- Handle
- @floydophone
Aug 21, 2023
Introducing Dagster Labs
In the spirit of simplification, the company formerly known as Elementl is now doing business as Dagster Labs.
- Name
- Nick Schrock
- Handle
- @schrockn
- Name
- Pete Hunt
- Handle
- @floydophone
Aug 18, 2023
Building an Outbound Reporting Pipeline
Learn how to use data engineering patterns and Dagster’s dynamic partitioning to build an outbound email report ...
- Name
- James Timmins
- Handle
- @jamestimmins
Aug 14, 2023
Parallel Computing on Dagster with Dask
Orchestrate your Dask computations and make your pipelines faster for larger data engineering and machine learning ...
- Name
- Odette Harary
- Handle
- @odette
Aug 11, 2023
Type Hinting in Python
In part VI of our Data Engineering with Python series, we explore type hinting functions and classes, and how type ...
- Name
- Elliot Gunn
- Handle
- @elliot
Aug 7, 2023
Environment Variables in Python
In part V of our series on Data Engineering with Python, we cover best practices for managing environment variables in ...
- Name
- Elliot Gunn
- Handle
- @elliot
Aug 3, 2023
Whats New in Data
Podcast: Data Orchestration, Dagster, and parallels to React.js
- Name
- Pete Hunt
- Handle
- @floydophone
Aug 3, 2023
Podcast: Drill to Detail - Dagster, Orchestration and Software-Defined Assets
Dagster Labs founder Nick Shrock is interviewed by Rittman Analytics founder Mark Rittman
- Name
- Nick Schrock
- Handle
- @schrockn
Aug 2, 2023
Podcast: The Scale Up Show - Interview with Pete Hunt
Ryan Staley interviewed Pete Hunt on how his experience at Facebook and Twitter is guiding his leadership of Dagster.
- Name
- Pete Hunt
- Handle
- @floydophone
Aug 1, 2023
Orchestrating dbt™ with Dagster
Orchestrate dbt with Dagster’s popular dbt integration, now with major enhancements to supercharge your dbt models as ...
- Name
- Rex Ledesma
- Handle
- @_rexledesma
- Name
- Sandy Ryza
- Handle
- @s_ryz
Jul 31, 2023
Speeding up the dbt™ docs by 20x with React Server Components
dbt docs slow? See how we dropped page load time and memory usage for a large dbt project by 20x using React Server ...
- Name
- Marco Salazar
- Handle
- @BkOptimism
- Name
- Pete Hunt
- Handle
- @floydophone
Jul 24, 2023
Podcast: A Geek Leader - Interview with Pete Hunt
John Rouda interviewed Pete Hunt, CEO of Dagster Labs, on React.js, open source and data orchestration.
- Name
- Pete Hunt
- Handle
- @floydophone
Jul 21, 2023
Announcing Dagster 1.4: Material Girl
The latest release brings major new dbt capabilities, new asset materialization controls, and more.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Jul 6, 2023
Video: Asset-Based Data Orchestration (from Data + AI Summit)
An overview of Dagster's asset-based orchestration approach, with data freshness sensors to trigger pipelines.
- Name
- Sandy Ryza
- Handle
- @s_ryz
Jul 5, 2023
LLM training pipelines with Langchain, Airbyte, and Dagster
This tutorial shows you how to combine Langchain, Airbyte, and Dagster to build maintainable and scalable pipelines for ...
Jun 26, 2023
Introducing Two New Self-Serve Plans for Dagster Cloud
'Solo' and 'Team' plans, with event-based pricing, will replace the old compute-duration based plan. We explain why we ...
- Name
- Pete Hunt
- Handle
- @floydophone
Jun 22, 2023
Revisiting the Poor Man’s Data Lake with MotherDuck
See how much easier you can collaborate using DuckDB’s high-powered cloud version MotherDuck to build a one-system data ...
- Name
- Pete Hunt
- Handle
- @floydophone
Jun 15, 2023
The Dagster Master Plan
Elementl CEO Pete Hunt shares the three priorities that guide how we will evolve Dagster.
- Name
- Pete Hunt
- Handle
- @floydophone
Jun 6, 2023
Backfills in Data & Machine Learning: A Primer
A step-by-step guide to using backfills and partitions to make data management more simple for data & ML engineers.
- Name
- Sandy Ryza
- Handle
- @s_ryz
May 31, 2023
Podcast: Data Platform Podcast - Orchestration & Psychology featuring Pete Hunt
Jason and Iva are joined by Pete Hunt, CEO of Elementl, to discuss orchestration tools and the psychology of companies.
- Name
- Pete Hunt
- Handle
- @floydophone
May 24, 2023
Elementl Raises $33 Million in Series B Funding to Accelerate Data Orchestration and Unleash Advanced Data Use Cases
The new capital will accelerate the development and adoption of Dagster, the open-source, cloud-native data ...
May 24, 2023
Dagster and the Decade of Data Engineering
We are pleased to announce Elementl's $33M Series B and share our vision for what's next for Dagster and the practice ...
- Name
- Nick Schrock
- Handle
- @schrockn
May 23, 2023
Building Better Analytics Pipelines
A recap of our live event on the benefits and techniques for orchestrating analytics pipelines.
- Name
- Pete Hunt
- Handle
- @floydophone
- Name
- Yuhan Luo
- Handle
- @yuhan
May 19, 2023
Introducing Dynamic Definitions for Flexible Asset Partitioning
Dagster’s dynamic partition definitions allow engineers to use the power of partitions in a broader range of scenarios.
- Name
- Claire Lin
- Handle
- Name
- Sandy Ryza
- Handle
- @s_ryz
May 17, 2023
Deciphering Arcane Kubernetes and ECS Errors with Dagster
Recent enhancements allow Dagster to surface clearer and more actionable errors to accelerate your development cycles.
- Name
- Daniel Gibson
- Handle
May 16, 2023
Config Systems: Airflow and Dagster
Contrasting the Airflow and Dagster configuration systems by rewriting the Airflow Slack Integration.
- Name
- Joe Van Drunen
- Handle
May 9, 2023
How to Maintain High Product & Code Quality As Your Startup Scales
Raising the quality bar requires process adjustments and a cultural shift.
- Name
- Bosmat Eldar
- Handle
- @bosmat
Apr 26, 2023
Announcing Dagster 1.3: Smooth Operator
Dagster 1.3 officially inducts Pythonic Config and Resources and brings new enhancements to Software-Defined Assets, ...
- Name
- Yuhan Luo
- Handle
- @yuhan
Apr 21, 2023
Case Study: Catalyst Cooperative - Liberating Public Utility Data with Dagster
The PUDL Project cleans and distributes analysis-ready energy system data to climate advocates, researchers, ...
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Apr 14, 2023
From Python Projects to Dagster Pipelines
In part IV of our series, we explore setting up a Dagster project, and the key concept of Data Assets.
- Name
- Elliot Gunn
- Handle
- @elliot
Apr 10, 2023
Case Study: Empirico - Enabling Large-scale, Multi-cloud Computing with Dagster
Abstracting away infrastructure concerns in large-scale computing with conditional multi-cloud processing.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Apr 4, 2023
Orchestrate Meltano Jobs with Dagster
Meltano provides 550 connectors and tools, all of which can be configured and orchestrated straight from Dagster.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Apr 3, 2023
Community Memo: Pythonic Config and Resources
Major ergonomic improvements are coming to Dagster's config and resources systems, including a Pydantic frontend.
- Name
- Nick Schrock
- Handle
- @schrockn
- Name
- Ben Pankow
- Handle
Mar 21, 2023
Best Practices in Structuring Python Projects
We cover 9 best practices and examples on structuring your Python projects for collaboration and productivity.
- Name
- Elliot Gunn
- Handle
- @elliot
Mar 20, 2023
Partitions in Data Pipelines
Partitioning is a technique that helps data engineers and ML engineers organize data and the computations that produce ...
- Name
- Sandy Ryza
- Handle
- @s_ryz
Mar 16, 2023
Tracking the Fake GitHub Star Black Market with Dagster, dbt and BigQuery
It's easy for an open-source project to buy fake GitHub stars. We share two approaches for detecting them.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
- Name
- Yuhan Luo
- Handle
- @yuhan
Mar 9, 2023
Announcing Dagster 1.2: Formation
Enhanced partitioned asset support and the introduction of Pythonic config and resources, and integration updates.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Mar 7, 2023
How Dagster Deploys 5X Faster with Warm Docker Containers
Using pex, Serverless Dagster Cloud now deploys 4 to 5 times faster by avoiding the overhead of building and launching ...
- Name
- Shalabh Chaturvedi
- Handle
Mar 6, 2023
Python Packages: a Primer for Data People (part 2 of 2)
An introduction to managing Python dependencies and some virtual environment best practices.
- Name
- Elliot Gunn
- Handle
- @elliot
Mar 6, 2023
Python Packages: a Primer for Data People (part 1 of 2)
The foundation of a solid Python project is mastering modules, packages and imports.
- Name
- Elliot Gunn
- Handle
- @elliot
Feb 28, 2023
Dagster Integrations Update
Dagster offers 47 integrations to accelerate your development, and we are working hard to expand and enhance them.
- Name
- Rex Ledesma
- Handle
- @_rexledesma
Feb 8, 2023
Migrating from Airflow to Dagster is now a Breeze
The newly released `dagster-airflow` library has made migrating off legacy Airflow and onto Dagster much easier.
- Name
- Joe Van Drunen
- Handle
Jan 9, 2023
Build a GitHub Support Bot with GPT3, LangChain, and Python
In this tutorial, we tap into the power of OpenAI's ChatGPT to build a GitHub support bot using GPT3, LangChain, and ...
- Name
- Pete Hunt
- Handle
- @floydophone
Dec 22, 2022
Converting an ETL Script to Software-Defined Assets
Lets talk about moving from an ETL script to a robust Dagster pipeline using Software-Defined Assets.
- Name
- Pete Hunt
- Handle
- @floydophone
Dec 16, 2022
Bringing Declarative Scheduling to dbt with Dagster
Declarative Scheduling takes the orchestration of dbt models as part of a larger pipeline to an entirely new level.
- Name
- Sean Lopp
- Handle
- @lopp
Dec 14, 2022
Announcing Dagster 1.1: Thank U, Next
A major release with Declarative Scheduling, multi-asset scheduling, and SDA partitioning. Plus Secrets management, ...
- Name
- Sandy Ryza
- Handle
- @s_ryz
Dec 8, 2022
Declarative Scheduling for Data Assets
Keep data assets up-to-date and determine whether source data has changed with declarative asset-based scheduling.
- Name
- Sandy Ryza
- Handle
- @s_ryz
Dec 7, 2022
Evaluating Dagster for Better Skiing - and a New Job
How quickstart projects snowball into new careers. A common data PoC walkthrough with Dagster.
- Name
- Sean Lopp
- Handle
- @lopp
Dec 1, 2022
Podcast: Build More Reliable Machine Learning Systems
Sandy Ryza explains how his background in machine learning has informed his work on the Dagster project.
- Name
- Sandy Ryza
- Handle
- @s_ryz
Nov 30, 2022
Getting Stuff Done: a Guide to Productive Software Engineering
To be a more productive software engineer you need to master changes, how these affect the program and others on the ...
- Name
- Alex Langenfeld
- Handle
- @alex_langenfeld
Nov 21, 2022
Safe and Easy: Managing Secrets in Dagster Cloud
Dagster Cloud’s new Environment Variables UI makes it easy to set up scoped environment variables.
- Name
- Erin Cochran
- Handle
- Name
- Daniel Gibson
- Handle
Nov 18, 2022
My Path to Elementl - Part 2
Pete Hunt takes over as CEO as Nick Schrock takes on the CTO role.
- Name
- Pete Hunt
- Handle
- @floydophone
Nov 11, 2022
Pushing REST-API data to Google Sheets with Dagster
A total beginners tutorial in which we store REST API data in Google Sheets and learn some key abstractions.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Nov 7, 2022
Adding Types to a Large Python Codebase
What we learned when we introduced dynamically typed code to a large Python codebase, bringing Dagster's public API to ...
- Name
- Sean Mackesey
- Handle
Oct 31, 2022
Orchestrating Machine Learning Pipelines with Dagster
How to use Dagster’s open source data orchestrator to build machine learning pipelines and train ML models.
- Name
- Sandy Ryza
- Handle
- @s_ryz
Oct 27, 2022
Case Study: Orchestrating Data Science at Zephyr AI
Zephyr AI applies data science to massive datasets of DNA and healthcare records to deliver novel AI-driven insights.
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Oct 25, 2022
Build a poor man’s data lake from scratch with DuckDB
DuckDB is so hot right now. Learn how to build a data lake from dbt using DuckDB for SQL transformations, along with ...
- Name
- Pete Hunt
- Handle
- @floydophone
- Name
- Sandy Ryza
- Handle
- @s_ryz
Oct 19, 2022
The Unreasonable Effectiveness of Data Pipeline Smoke Tests
Data practitioners waste time writing unit tests to catch bugs they could have caught with smoke tests.
- Name
- Sandy Ryza
- Handle
- @s_ryz
Oct 17, 2022
Web Workers are not the Answer
A tale of overstretched logs, counterintuitive web worker behavior, and ultimately a troublesome cursor issue.
- Name
- Marco Salazar
- Handle
- @BkOptimism
- Name
- Alex Langenfeld
- Handle
- @alex_langenfeld
Oct 16, 2022
Dagster at all 5 Steps of the Development Lifecycle
Dagster facilitates a data engineers work across all five steps in the development lifecycle.
Oct 6, 2022
A Dagster Crash Course
If you are looking to get up and running with Dagster in 10 minutes or less, this is a good place to start. Buckle up.
- Name
- Pete Hunt
- Handle
- @floydophone
Oct 4, 2022
Postgres: a Better Message Queue than Kafka?
When lots of event logs must be stored and indexed, Kafka is the obvious choice. Naturally, our queue runs on Postgres.
- Name
- Pete Hunt
- Handle
- @floydophone
Aug 24, 2022
Case Study: How EvolutionIQ Rebuilt its ML Platform for Enormous Productivity.
A guide for CIOs/CTOs and engineering leaders looking to master the Modern Data Stack and develop a high performance ...
- Name
- Fraser Marlow
- Handle
- @frasermarlow
Aug 17, 2022
Spend Less Time Debugging with Dagster
It’s not uncommon for a data engineer to devote 80% of their day to debugging. Dagster radically improves on this.
- Name
- Sandy Ryza
- Handle
- @s_ryz
- Name
- Owen Kephart
- Handle
Aug 9, 2022
Launching Dagster Cloud to GA
The enterprise orchestration platform that puts developer experience first: hybrid or serverless deployments, native ...
- Name
- Nick Schrock
- Handle
- @schrockn
Aug 5, 2022
Introducing Dagster 1.0: Hello
Announcing Dagster 1.0. - a stable foundation for building the orchestration layer for modern data platforms.
- Name
- Sandy Ryza
- Handle
- @s_ryz
Aug 3, 2022
The Open Core Business Model
The relationship between Dagster, the open-source project, and Dagster Cloud, our hosted SaaS platform.
- Name
- Nick Schrock
- Handle
- @schrockn
Jul 26, 2022
Dagster Cloud goes SOC 2
Elementl, the company behind the Dagster data orchestration tool achieves SOC2 compliance.
- Name
- Selina Li
- Handle
Jul 25, 2022
Dagster Day: Announcing Dagster 1.0 and Dagster Cloud
The release of Dagster 1.0 and the GA launch of Dagster Cloud represent major milestones in the evolution of our ...
- Name
- Nick Schrock
- Handle
- @schrockn
Jul 12, 2022
Roman Roads in Data Engineering: Don't Write Data Pipelines from Scratch
Work in a way that lays the foundation for your next data product while you're building your current one.
- Name
- Claire Lin
- Handle
- Name
- Sandy Ryza
- Handle
- @s_ryz
Jun 23, 2022
Podcast: The Data Exchange - Software-defined Assets
Nick Schrock on software-defined assets, a new approach to managing, maintaining, and orchestrating data declaratively.
- Name
- Nick Schrock
- Handle
- @schrockn
Jun 22, 2022
My Path to Elementl: Pete Hunt
Pete Hunt discusses what caused him to make the leap from Twitter to Elementl.
- Name
- Pete Hunt
- Handle
- @floydophone
Jun 20, 2022
Orchestrating Python and dbt with Dagster
How asset-focused orchestration bridges the gap between some of data's most popular tools.
- Name
- Owen Kephart
- Handle
Jun 15, 2022
Dagster 0.15.0: Cool for the Summer
In 0.15.0, software-defined assets are now marked fully stable and are ready for primetime.
- Name
- Mollie Pettit
- Handle
Mar 9, 2022
New in 0.14.0: Dagster-Airbyte Integration
0.14.0 introduces a deep integration with Airbyte: view Airbyte logs directly in Dagit, and every updated table will be ...
- Name
- Owen Kephart
- Handle
Mar 1, 2022
Introducing Software-Defined Assets
Software-Defined Assets are a new abstraction that allows data teams to focus on the end products, not just the ...
- Name
- Sandy Ryza
- Handle
- @s_ryz
Mar 1, 2022
Announcing Dagster 0.14.0: Table Schema API + Pandera Integration
Introducing two asset observability-enhancing features: Table Schema API, and an integration with the dataframe ...
- Name
- Sean Mackesey
- Handle
Mar 1, 2022
Announcing Dagster 0.14.0: Never Felt Like This Before
We’re thrilled to release version 0.14.0 of Dagster. This version introduces much more mature version of ...
- Name
- Mollie Pettit
- Handle
Feb 17, 2022
Rebundling the Data Platform
'The Unbundling of Airflow' argued that modern data stack solutions (data ingestion, data transformation, reverse ETL) ...
- Name
- Nick Schrock
- Handle
- @schrockn
Dec 2, 2021
Introducing Dagster Cloud
Dagster Cloud, the enterprise orchestration platform that puts developer experience first, with fully serverless or ...
- Name
- Nick Schrock
- Handle
- @schrockn
Nov 20, 2021
Podcast: Laying the Foundation of your Data Platform for the Era of Big Complexity
Listen to founder and CEO Nick Schrock talk about how Dagster helps tame the complexity and scale when working with ...
- Name
- Nick Schrock
- Handle
- @schrockn
Nov 17, 2021
Podcast: Hello Big Complexity: Is Your Modern Data Stack Ready?
Listen to Nick Schrock discuss the evolution of data from Big Data to Big Complexity in this episode of the Mad Data ...
- Name
- Nick Schrock
- Handle
- @schrockn
Nov 16, 2021
Why Elementl and Dagster: The Decade of Data
Announcing our $14M Series A led by Index Ventures, alongside Sequoia Capital, Slow Ventures, Coatue, Amplify Partners, ...
- Name
- Nick Schrock
- Handle
- @schrockn
Nov 8, 2021
New in Dagster 0.13.0: Logging Improvements!
Logging without context, instance-wide handlers, capturing python logs, and more! Learn about the improvements we've ...
- Name
- Owen Kephart
- Handle
Oct 28, 2021
Announcing Dagster 0.13.0: A New Foundation
We’re proud to announce 0.13.0 of Dagster with dramatic improvements to our core APIs, completely revamped UI, and ...
- Name
- Nick Schrock
- Handle
- @schrockn
Aug 10, 2021
Community Memo: Moving Dagster's Core APIs Towards 1.0
Dagster commits to a stable set of production-ready APIs for building solid data platforms.
- Name
- Sandy Ryza
- Handle
- @s_ryz
Jul 19, 2021
Announcing Dagster 0.12.0: Into the Groove
In 0.12.0, we introduce pipeline failure sensors, solid-level retries, and more convenient testing APIs.
- Name
- Owen Kephart
- Handle
May 25, 2021
Community Memo: Approachability Improvements
In the last two months, we've made a set of changes aimed at making Dagster more approachable: to smooth out its ...
- Name
- Sandy Ryza
- Handle
- @s_ryz
May 18, 2021
Case Study: Incrementally Adopting Dagster at Mapbox
At Mapbox, we've adopted Dagster without breaking compatibility with our legacy Airflow systems -- and with huge gains ...
- Name
- Ben Pleasanton
- Handle
May 13, 2021
Moving past Airflow: Why Dagster is the Next-generation Data Orchestrator
A comparison between Dagster and Airflow. Here we detail the differences between the two systems, and make the case for ...
- Name
- Nick Schrock
- Handle
- @schrockn
Apr 1, 2021
Announcing Dagster 0.11.0: Lucky Star
In 0.11.0, we introduce dynamic orchestration, a new backfill UI, and support for tracking asset lineage.
Jan 19, 2021
Announcing Dagster 0.10.0: The Edge of Glory
In 0.10.0, we introduce unique event-based scheduling capabilities, hardened deployments on Kubernetes, and new ...
- Name
- Nick Schrock
- Handle
- @schrockn
- Name
- Max Gasner
- Handle
Dec 9, 2020
Case Study: Good Data at Good Eggs - Using Dagster to Manage the Data Platform
Running pipelines is only part of running a data platform. We need to manage the platform and control technical debt. ...
- Name
- David Wallace
- Handle
- @davidjwallace
Nov 5, 2020
Case Study: Good Data at Good Eggs - Data Observability with the Asset Catalog
Dagster gives us a single "pane of glass" for data assets. Analysts can look up when a Stitch raw data ingest occurred, ...
- Name
- David Wallace
- Handle
- @davidjwallace
Oct 29, 2020
Dagster and dbt: Better Together
People sometimes ask us — should I use Dagster, or should I use dbt? We view Dagster and dbt as complementary ...
- Name
- AJ Nadel
- Handle
- @AJ_Nadel
- Name
- Bob Chen
- Handle
Oct 1, 2020
Case Study: Good Data at Good Eggs - Data Infrastructure Correctness and Reliability
Dagster’s custom data types helped achieve correctness and reliability in our data ingest process, less downstream ...
- Name
- David Wallace
- Handle
- @davidjwallace
Oct 1, 2020
Case Study: Good Data at Good Eggs - Part 1 of 4
Adopting Dagster transformed our data platform team. We hope our experience is encouraging to other teams facing ...
- Name
- David Wallace
- Handle
- @davidjwallace
Sep 16, 2020
Testing and Deploying PySpark Jobs with Dagster
Spark has a beautiful API but developing with it is a pain because different stages of development and deployment ...
- Name
- Sandy Ryza
- Handle
- @s_ryz
Sep 15, 2020
Community Memo: September 2020 Update
A retrospective of our 0.9.0 release, a preview of our 0.10.0 roadmap, and Prezi's journey from a homegrown ...
Sep 10, 2020
New Integration: Great Expectations for Dagster
We’re thrilled to announce a new integration between Dagster and a fellow open-source project, Great Expectations (GX).
- Name
- Leor Fishman
- Handle
Aug 25, 2020
Podcast: Forward Thinking Leaders - How to Sell New Tech Concepts to Developers
Nick Schrock shares insights on how to on how to sell new tech concepts to developers.
- Name
- Nick Schrock
- Handle
- @schrockn
Aug 11, 2020
Dagster: The Data Orchestrator
As a workflow engine, Dagster moves beyond ordering and executing data computations. It introduces a new primitive: a ...
- Name
- Nick Schrock
- Handle
- @schrockn
- Name
- Max Gasner
- Handle
Feb 26, 2020
Announcing Dagster 0.7.0: Waiting To Exhale
With 0.7.0 we set out improve the Dagster experience with large, production-scale pipelines, deployable to Kubernetes.
Oct 10, 2019
Announcing Dagster 0.6.0: Impossible Princess
Dagster 0.6.0 comes “batteries-included” and pluggable options to execute, monitor, schedule, deploy, and debug your ...
Jul 8, 2019
Introducing Dagster
Elementl announces an early release of Dagster, an open-source library for building ETL processes, ML pipelines and ...