Zero Incidents, Maximum Velocity: How HIVED achieved 99.9% pipeline reliability with Dagster

KEY RESULTS:

• 99.9%+ pipeline reliability: Zero data incidents over 3 years supporting millions of deliveries
• 3-7 Day Turnaround: New data products and analytics requests delivered within a week for rapid response to business needs
• Fully automated pipelines: Used Dagster to eliminate manual processes, transforming legacy processes into fully automated pipelines with dependency management and failure monitoring
• Self-service capabilities: Analytics engineers can independently add new data sources through YAML configurations.

The challenge

When Sebastián Arriagada joined HIVED three years ago, the UK logistics company had built a pragmatic data solution that worked well for its stage. The team leveraged reliable ingestion tools, such as Fivetran, alongside other legacy processes for BigQuery ingestion. Scheduled BigQuery queries handled their core analytics needs through straightforward cron jobs. This approach had served them well as they established their data foundation and built initial reporting capabilities.

However, as HIVED scaled rapidly and operational decisions became increasingly dependent on dashboard insights, the limitations of their current approach became apparent. The system lacked dependency management between processes, meaning failures could cascade without clear visibility into downstream impact. 

For Sebastián, the challenge wasn't that their existing system was wrong, but that HIVED had outgrown it. As an analytics-heavy organization where operational decisions depend on reliable data, they needed the visibility into data lineage, proactive failure handling, and dependency management that their current architecture couldn't provide. The question wasn't whether to replace what they had built, but how to evolve it to match their current operational needs.

The company needed modern data orchestration to establish dependency management, failure monitoring, and end-to-end visibility across its data pipelines. As a single-person data engineering team supporting a team of analytics engineers and growing operational demands, Sebastián required tools that were reliable, easy to maintain, and could scale without constant intervention, moving from reactive firefighting to proactive platform management that would keep HIVED delivering for its customers, flawlessly and on time.

The solution

Sebastián had used Airflow previously and found that Airflow’s limited local development and testing capabilities made building data pipelines slow and cumbersome. Dagster's developer-centered approach, including local testing capabilities, built-in monitoring, and reliable scheduling, made it the clear choice for a lean team that needed tools requiring minimal maintenance overhead.

The implementation centered on creating a unified view of data operations with native dependency management and failure monitoring. HIVED built their platform using open-source Dagster deployed on Kubernetes, establishing automated pipelines with proper lineage tracking and Slack notifications for failures. With the goal of replacing reactive maintenance with proactive development, Sebastián developed an abstraction layer that allows analytics engineers to define new data ingestions through YAML configurations, enabling self-service capabilities without requiring constant data engineering intervention. 

HIVED ultimately used Dagster as the orchestration layer for a unified data stack that includes Airbyte for ingestion, dbt for transformations, and BigQuery as their warehouse. This integrated platform granted immediate visibility into data lineage and impact analysis through Dagster's UI, eliminating the blind spots that plagued their previous cron-based approach. Now, when failures occur, the team can instantly see which downstream models and dashboards are affected, enabling proactive communication with their data end users — and earning back their trust. This architectural foundation positioned HIVED's analytics capabilities alongside their expanding logistics network across the UK.

“One of the main things I really like about Dagster is how local development and testing were taken into account from the very beginning...How you just take your schedule and your assets and do all that work locally, then spin up Dagster and test the whole thing very easily."

Because I'm a one-man data platform team, I need tools that are easy to set up, easy to maintain in general, and that don't require me every day to go fix this and that. From both a developer experience and reliability perspective, it was an easy choice really to go with Dagster. 

The results

HIVED has achieved remarkable reliability over three years of Dagster implementation, with Sebastián noting, "I don't think we have had any incident with Dagster in these three years. Really." This 99.9%+ pipeline reliability supports millions of deliveries without data incidents and has fundamentally transformed how HIVED’s business users interact with analytics and even the most experienced new employees are consistently surprised about the quality and reliability of data at HIVED.

Their new Dagster-centered data platform also unlocked dramatic improvements in team velocity and responsiveness to data end-user requests and business needs. New data products and analytics requests now have a turnaround time of just three to seven days, depending on complexity, compared to the previous firefighting mode, where infrastructure maintenance consumed most engineering cycles. HIVED’s analytics engineers work full-time on new dashboards and business intelligence rather than troubleshooting broken pipelines, so they can make critical business decisions around HIVED’s operational expansion across the UK market.

Operationally, the unified orchestration platform enables sophisticated analytics that power business growth, including location expansion analysis using integrated mapping tools and delivery performance optimization. To democratize data access, Sebastián built a self-service YAML-based ingestion system that allows analytics engineers to independently add new data sources through pull requests, removing bottlenecks (and lightening his own load) while maintaining governance and Data engineering best practices. This infrastructure efficiency, combined with open-source tooling deployed on Kubernetes, keeps costs lean while powering a logistics operation that relies heavily on real-time operational metrics for route optimization and performance measurement.

Looking ahead

HIVED’s immediate roadmap focuses on modernizing their remaining legacy processes and expanding platform capabilities. Sebastián plans to migrate older ops-based pipelines to Dagster's asset-centric approach to leverage enhanced UI features and improved visibility. The team is also implementing Dagster’s Looker integration to complete their end-to-end data lineage visibility, providing a comprehensive view from ingestion through business intelligence consumption.

HIVED is looking to move existing ML and AI workflows onto a unified (orchestration) platform to support its growing Data Science and AI team.  This will include ongoing training and deploying models for route optimization, delivery time predictions, and operational efficiency improvements. Sebastián says HIVED’s Dagster-centered data infrastructure allows the organization to seamlessly integrate ML workflows alongside its existing analytics pipelines. 

Sebastián’s advice to other high-growth companies in the logistics and operations sectors: think orchestration first. "I feel like the orchestrator is such a core piece in the whole data stack, and that is where you should start,” he explains. “It also influences the decisions around the other tools that you're going to use." As HIVED continues expanding their UK coverage, the scalable platform architecture ensures they can support growing operational complexity without proportional increases in data engineering overhead, maintaining their lean infrastructure approach while giving their end users trustworthy data for the sophisticated analytics HIVED needs to drive the company’s growth.

Key takeaways

• 99.9%+ pipeline reliability: Zero data incidents over 3 years supporting millions of deliveries, compared to frequent failures with the previous cron-based approach
• 3-7 day turnaround: New data products and analytics requests delivered within a week, enabling rapid response to business needs
• Fully automated pipelines: Used Dagster to eliminate manual processes, transforming early-days paid ingestion solutions or legacy processes to fully automated pipelines with dependency management and failure monitoring
• Team velocity transformation: Shifted from firefighting mode to building new data products, with all analytics engineers working full-time on business intelligence rather than troubleshooting
• Self-service capabilities: Analytics engineers can independently add new data sources through YAML configurations and pull requests, removing data engineering bottlenecks
• Cost-effective infrastructure: Lean open-source stack on Kubernetes supports millions of deliveries while maintaining minimal operational overhead for a single-person data engineering team

‍