How Magenta Telekom Built the Unsinkable Data Platform

Magenta Telekom, a leading Austrian telecom provider and subsidiary of Deutsche Telekom, serves over 5 million customers with mobile, internet, and TV services. To modernize its data infrastructure and reduce reliance on manual workflows, Magenta adopted Dagster as the core of its new data platform. With Dagster, Magenta is building a modular, scalable data platform that enables secure, observable, and efficient data operations across teams - laying the foundation for Data- and AI-driven decision-making that delights Magenta customers and drives above-average revenue growth to outpace the market.

• Unification: Siloed data in Excel → Centralized, observable data domains
• Standardization: Bespoke scripting and shadow IT -> Blueprint-based modular platform
• Governance: Catalog chaos and debate -> Declarative, enforceable domain standards
• Onboarding: 3 months -> 1 day

Modernizing a legacy data system

Magenta  is a company built on data. From streaming video services to core telco infrastructure and billing systems, Magenta's business depends on data flowing reliably and safely across dozens of teams, tools, and services.

For years, Magenta’s data environment was loosely connected and complex. Many critical workflows were managed through Excel sheets shared over email, and two-thirds of engineering efforts were dedicated to maintaining legacy systems. Shadow IT had naturally emerged in some areas, and without a unified governance model, internal data catalogs developed independently rather than in coordination.

This case study uncovers how Magenta turned those challenges into an opportunity: reimagining its data platform from the ground up. With Dagster as the orchestration backbone, Magenta has built a modular, scalable architecture that empowers developers, accelerates onboarding, and sets the stage for long-term innovation and AI readiness.

Designing a platform to scale with the business

Magenta set out to overhaul its data infrastructure, not just to modernize it, but to re-architect how data is developed, governed, and used across the company. Legacy systems had drained engineering resources, while disconnected tooling and manual processes slowed innovation and introduced risk. Magenta needed a platform that could bring visibility into all data work, eliminate shadow IT, automate error-prone workflows, and enable both security and speed at scale. They weren’t just looking for a better scheduler; they were building a composable, secure, developer-first platform that could grow with their business.

What Magenta needed in a data orchestration platform:

• Free up engineering time from legacy maintenance: A modern platform with automation and standardized blueprints that reduces manual upkeep and enables teams to focus on net-new development.
• Replace manual Excel workflows used in critical processes: End-to-end data pipelines that automate ingestion, validation, and reporting, as well as support business logic in reusable, tested components.
• Bring visibility to all data pipelines, including those built by analysts or in the shadows: A unified orchestration layer with full asset lineage, observability, and support for both engineering and business user workflows.
• Resolve governance disagreements and enforce standards without friction: A declarative architecture that embeds governance (like testing, lineage, and metadata capture) directly into the development lifecycle.
• Onboard new contributors, including offshore consultants, without months of setup: Pre-configured, containerized environments with access controls and network setup out of the box, so developers could start building from day one.
• Support secure, scalable experimentation with AI and machine learning: Environments that separate production from experimentation, with guardrails like data masking, role-based access control, and reproducible infrastructure for sensitive data work.

Why Magenta chose Dagster

What we want to build is something that is like an unsinkable ship… compartmentalized on its own, with adjacent components for things that would traditionally have floated off into shadow IT.
 - Georg Heiler, Senior Data Expert, Magenta

The Magenta data enablement team has been satisfied with having Dagster in their stack, saying, “It includes the quality components, and it’s just so much smoother, end-to-end.” The team also appreciates that Dagster provides:

• Composable, modular architecture:  Magenta needed a platform that allowed them to define standardized data "domains:” isolated, self-contained, and deployable units that teams could build and operate independently. Dagster's project structure and support for code locations and multi-project setups aligned perfectly with this need.
• Developer-first experience: Magenta’s data enablement team wanted tools that felt like modern software engineering: with CI/CD pipelines, testability, and low-friction local development. Dagster’s focus on developer experience, including testing, observability, and asset lineage, was a major draw.
• Declarative, asset-based orchestration: They were drawn to Dagster’s asset-centric model, which organizes workflows around data outputs instead of just tasks or steps. This made governance easier, improved traceability, and simplified debugging.
• Support for secure, scalable environments: With strict compliance and data security requirements, Magenta needed to isolate production workloads from development and enable safe collaboration with external teams. Dagster’s code location abstraction and ability to separate orchestration from execution helped enable this.
• Observability and governance embedded in the platform: Dagster’s built-in asset lineage, testing, and metadata features provide Magenta a foundation for improved governance without slowing teams down or adding yet another tool to manage.
• Integration flexibility: Magenta needed a platform that supports Spark, REST APIs, Tableau dashboards, and eventually machine learning workflows. Dagster’s flexible plugin system, support for custom IO managers, and integrations like dbt and Tableau made it a natural fit.

Dagster’s architecture gave them confidence that they could extend the platform over time to support new use cases (like LLMs, secure AI workflows, or external data products).

This is not just a data warehouse… this is a platform built to scale with the business.

‍
Magenta did more than just adopt Dagster: they designed an entire platform around it. The team adopted a blueprint-driven approach, combining best practices with the necessary flexibility for large-scale enterprise data environments. Their goal was to create a composable, secure, and self-serve ecosystem where developers and analysts alike could build, deploy, and manage data workflows with confidence.

Blueprint-based data domains: At the heart of Magenta’s implementation is the concept of standardized data domains. Each domain, whether for network planning, billing, or analytics, is instantiated from a shared blueprint that includes:

• A Docker container for reproducible runtime environments
• Dagster Code locations for orchestration and asset lineage
• dbt project for each domain, SQL-based modeling, and data testing using this project to give a dbt cloud-like experience with dbt core. 
• GitLab CI/CD pipelines for continuous integration and delivery
• Pre-configured Python libraries and utility scripts. Georg Heiler has discussed this topic here and provides a starter template for those who want to try it hands-on. 
• Infrastructure-level data domain isolation using cloud-native security boundaries (i.e., in Azure subscriptions, in GCP projects, and in AWS accounts).
• Shared logic for data ingestion, validation, and output formatting
• Standardized integrations with VertexAI, DuckLake, and Tableau

Domains are customized with cruft to fit their specific use case, but still benefit from centralized updates and automation. When the core blueprint evolves, improvements can be rolled out automatically to all domains.

GitLab Workspaces for day-one productivity: To streamline onboarding and collaboration, Magenta implemented secure, cloud-based GitLab Workspaces. These environments provide a turn-key solution for working with data. They:

• Come pre-loaded with the right packages, network permissions, and access controls and resources - even GPUs, this saves countless hours of troubleshooting by delivering a turn-key solution to work with data in Magenta. 
• Let internal developers and external consultants spin up a dev environment instantly, increasing productivity without sacrificing governance
• Eliminate the three to five-month ramp-up previously required for offshore contributors. This is achieved by
  
  • Provisioned infrastructure (all the tools are set up, licenses pre-prepared, and flexible resource configuration)
  • Firewalls pre-configured (jump host)
  • By utilizing Dagster’s built-in dependency injection to quickly enable less technical users. Using the right levels of abstraction (IO managers, resources) allows experts to design them, and others vibe code business logic. No longer are the limits of collaboration with business a long lead time to move to production. All critical and complex concerns are already handled and abstracted, allowing people to focus directly on working with the data. A bit similar to Outlook for emails.
• Support safe collaboration without compromising security or compliance

Integrated ingestion and transformation layers: Magenta built an abstraction layer for data ingestion, using Dagster Pipes to handle Spark-based workloads and embedded ELT for simpler use cases. This allowed them to standardize ingestion across domains, while still supporting:

• REST APIs and flat files
• Large-scale batch processing with Spark
• Lightweight pipelines for internal automation

‍

Transformation logic is handled using dbt, which is fully integrated into the Dagster asset graph. Analysts can contribute models directly via Git workflows, with automated testing and validation built in.

Data warehouse foundation with Data Vault modeling: Underlying Magenta's orchestration layer is a Data Vault approach that automates the mechanics of the warehouse itself. While Dagster orchestrates the overall data usage across the enterprise from ingestion through to business intelligence, the Data Vault methodology provides the resilient, auditable foundation for storing and historizing raw business data. This separation of concerns allows Dagster to focus on workflow orchestration and business logic, while Data Vault handles the structural integrity and scalability of the warehouse layer.

Tableau and BI automation: To close the loop from ingestion to insight, Magenta created a custom Tableau integration that allows Dagster to refresh only the dashboards affected by upstream data changes. This reduces costs and ensures faster, more relevant updates across the business intelligence layer.

Governance and security: Magenta embedded governance directly into their pipelines using Dagster’s declarative asset model and custom test libraries. Features include:

• Stateless and stateful data quality checks
• Custom dbt macros for business-specific rules
• Asset-based lineage tracking and auditing
• Metadata capture to support data catalog integrations with DataHub and Collibra. 

Security was also a top priority. Magenta architected its platform to separate production and development environments, enabling secure experimentation with masked and anonymized data, as well as strict role-based access control. 

From bottlenecks to breakthroughs

Within six months of implementation, Magenta has experienced measurable improvements across its data ecosystem from engineering velocity to business confidence in data. The platform hasn’t just modernized how workflows run; it’s reshaped how teams collaborate, build, and scale. With Dagster, Magenta now benefits from:

Faster time to value

With pre-configured development environments and blueprint-based domain templates, Magenta teams can now go from zero to shipping data pipelines in a matter of hours. This has reduced onboarding time from months to a single day, allowing internal and external teams to contribute immediately, focusing on business impact instead of being bogged down by infrastructure impediments.

Unified observability and lineage

Dagster’s asset-centric approach provides full visibility into data flows: from ingestion to reporting. Teams no longer rely on ad hoc scripts or guesswork to understand what’s happening. With built-in testing, lineage, data quality, and metadata, data flows smoothly through the system, enabling stakeholders to rely on accurate and timely information. Building this out is foundational work for enabling a semantic layer that will allow less technical users to chat with the data for true self-service analytics. 

Modular, scalable architecture

By organizing each use case as a separate domain, Magenta can scale its platform efficiently without increasing complexity. New domains automatically inherit shared standards and best practices, while platform updates can be deployed centrally across all domains. With many domains already in place, Magenta needed a solution that enables the organization to quickly provision new workspaces and manage access without requiring custom development each time.

Georg and Aleksander Milicevic recently gave a talk at the Vienna Data engineering meetup on this architecture and approach. You can check out the talk here.

Enterprise-grade governance

Separation between standardized developments and production environments ensures compliance, while new support for masked and anonymized data access enables secure experimentation with sensitive data. Magenta can now confidently allow analysts and external partners to collaborate within tightly scoped environments.

Reduced cost and errors

Replacing manual workflows, such as Excel-based billing & various SQL Scripts without a single source of truth, with automated, validated pipelines has improved operational efficiency and reduced the risk of human error. BI dashboards are now refreshed intelligently, enhancing data quality and keeping dashboards up to date. 

We were driving with a 30-year-old Fiat car, and now we have a Formula One racing car.

Whats Next

‍We would say this is our game-changing platform. Instead of onboarding for several months, we have day one productivity.

Future plans for Magenta and Dagster: with strong backing from its new Data Tribe Lead, Magenta aims to increase engineering investment in the new platform from recent 20–30% to over 80%, as the new GCP-based platform, Pluto, becomes the unified and converged data foundation that will position Magenta as a Data & AI powerhouse. Magenta also plans to roll out its first fully integrated, end-to-end commercial use cases that will showcase the complete data lifecycle - from ingestion to transformation to visualization - all powered by Dagster.

‍