How Vanta Eliminated Data Bottlenecks with Dagster

Vanta is a leader in trust management and compliance automation, empowering more than 10,000 companies to efficiently achieve and maintain critical security and compliance standards. Adopting Dagster as their modern data orchestration platform helped Vanta drastically reduce data delays, streamline their on-call operations, and significantly expand self-service for developers, data engineers, and analysts.

Dagster is a central piece of everything we do. Without it, we wouldn’t have been able to get to the level of self-service we are at today.  - Jake Peterson, Head of Data & Analytics, Vanta

Improving developer experience and on-call efficiency with a modern data orchestration platform

The compliance journeys of Vanta’s customers vary in complexity and often span months or years. Vanta uses data to support their customers throughout their journeys, as well as support internal teams including product, sales, and marketing with critical insights.

The Vanta data team is responsible for a large footprint, including supporting a fast-shipping engineering organization. With such a strong focus on delivery, it’s no surprise the data team sought out a solution to increase the efficiency and self-service capabilities of their data platform.

Challenges of a fragmented stack

Prior to Dagster, the team was experiencing the frustration that comes with cron scheduling across multiple data tools and systems. Their reliance on Snowflake, dbt, multiple ETL providers, and a growing number of additional data sources underscored the need for a central orchestration solution. Several key problems also inspired their search:

Lack of unified programming model. The absence of a unified programming model prevented the team from being able to adopt the software development practices needed to extend self-service capabilities to developers and analysts.

Frequent after hours escalations. At Vanta, data analysts also participate in on-call shifts, but diagnosing business-critical data errors required complicated runbooks outside of their immediate expertise, resulting in frequent escalations.

Lengthy troubleshooting. Alerts were scattered across multiple tools and troubleshooting data errors often required traversing 6 or more tools to find the root cause.

Difficulty adding new integrations. Vanta’s ETL providers offered limited integration support. The team frequently built custom integrations using AWS Step Functions–a process that took up to 2 weeks and added significant infrastructure complexity.

We were looking for an orchestration solution that could give us a single pane of glass for managing daily and on-call operations – and would allow anyone across the team to self-serve. - Jake Peterson

Why Vanta chose Dagster

After evaluating various solutions, Vanta selected Dagster for its unified programming model, central visibility, and ability to integrate with their existing data stack while providing a foundation for future growth. As heavy users of dbt, the dbt integration was especially important, and the team appreciated its near-instant setup.

We had slotted weeks to do our dbt integration with Dagster. And then we spun it up and it was one button and we were done. - Jake Peterson

The migration to Dagster was executed thoughtfully in three phases:

• Initial proof of concept and detailed planning
• Sunset legacy systems, minimizing operational disruptions
• Fully centralized orchestration, achieving a single pane of glass

The Dagster support team provided implementation support throughout, ensuring the transition went smoothly. Business users reported the migration went virtually unnoticed.

The excellent support was an incredibly pleasant surprise and has felt like a true extension to our team, especially during implementation. - Jake Peterson

The results of a modern data orchestration platform

Central management of a growing stack. Because Dagster sits at the center of their work, the data team can seamlessly support a growing number of pipelines, integrations, and data sources without losing central visibility and control.

Streamlined time to insight. Business data freshness significantly improved, cutting latency from 7 hours to less than 30 minutes for critical go-to-market systems like Salesforce.

We’ve moved business-critical data freshness from 7 hours down to 30 minutes or faster, significantly impacting decision-making speed for our go-to-market teams. - Jake Peterson

Minimized troubleshooting time. Previously, troubleshooting required navigating multiple tools and performing complex manual retries. Dagster centralized alerts and enabled single-click retries.

Tracing data issues used to involve clicking through 6 different tools. Now we can instantly rerun and check processes in Dagster, dramatically speeding up troubleshooting and getting insights to business partners sooner. - Jake Peterson

Reduced escalations. Dagster reduced operational burden by empowering data analysts to independently manage most incidents during on-call shifts, decreasing the escalation to senior engineers.

Our analytics team can now confidently manage on-call incidents, significantly reducing off-hours escalations. - Jake Peterson

Reduced time to new data sources and integrations. Dagster's out-of-the-box integrations were near-instant for the team to set up. Integrations that previously required extensive custom development and weeks or months of effort are now operational in a week or less with Dagster. Similarly, new data sources can be spun up as needed without losing visibility and control.

One of the biggest benefits of Dagster is being able to spin up new integrations quickly. We write Python code, test it quickly, and have immediate confidence it’ll work without worrying about boilerplate or infrastructure complexity. - Adam Barber, Data Engineering Manager, Vanta

Expanded self-service. Since implementing Dagster, the team has assisted with projects across the organization that would have taken twice as long and previously required a senior engineer. Now they can do the same projects in half the time with junior developers leading important initiatives.

We recently measured our engineering velocity—a major initiative. Without Dagster, it would’ve taken twice as long. Instead, a junior developer delivered it rapidly. The exact ROI is tough to quantify, but the productivity impact is enormous. - Jake Peterson

Decreased spend and reliance on legacy tools. Dagster now handles all of the extract and load for data at Vanta, giving them greater control and visibility than they had through dedicated ETL providers. Additionally, they were able to save substantial spend by migrating from dbt Cloud to Dagster.

Why use an ETL provider when we get no insight into what's going on? We can run it through Dagster with our own code–and test it–and have complete visibility throughout the process. - Adam Barber

Improved adoption of best practices. Dagster’s local development experience and support for software development workflows has enabled the team to improve adoption of best practices like unit testing and branch deployments that were previously hindered by their stack. Additionally, they’ve been able to replace their cron-based system with declarative workflows and sensors.

With Dagster, we've moved away from cron to declarative orchestration where things run when they need to run and when the data is ready. - Jake Peterson

Future opportunities

Encouraged by their success with Dagster, Vanta plans to expand the possibilities enabled by the platform, including automated testing in CI. They’re also working closely with the team to explore deeper integration with Sigma, particularly for high-traffic workbooks, and OpenAI.

‍