From scattered scripts to unified platform: How Dagster powers data decisions for 4.5 million Belgian citizens

As one of Belgium’s three linguistic communities, the Fédération Wallonie-Bruxelles oversees education, sports, culture, youth services, etc. for 4.5 million citizens — and manages a workforce of over 7000, spread across 5 different general administrations (“Administrations Générales”) and hundreds of teams/departments within the ministry. With such massive scope, the organization generates staggering amounts of data from the citizens it serves and the organizations it subsidizes. But FWB’s fragmented, closed-source data systems were slow, inaccessible, and too often maddeningly manual.

Martin Erpicum is a data engineer at Fédération Wallonie-Bruxelles, where he helped transform FWB’s cumbersome legacy system into a modern, modular data platform. As Martin explains to his C-suite, Dagster’s flexibility enables his team to meet FWB’s significant organizational needs without being constrained by the platform. “No vendor lock-in means that 90% of our code can be re-used anywhere,” he says. “And, in just ten months of use, I’ve already seen multiple instances where Dagster’s data observability features helped us catch and resolve issues that would have otherwise gone unnoticed.”

Key Results

• 2x pipeline speed: Time to deliver results from raw data to BI tools reduced by more than 50%
• Reactive -> proactive: Data teams now proactively propose new data products to users, instead of just reacting to requests 
• Visibility unlocked: Data quality issues that would previously go undetected are now quickly discovered and resolved with Dagster’s unified control plane
• Zero accessibility -> self-service: End users now provision their own data dashboards from always up-to-date data mart

Breaking free from closed-source constraints

When Martin arrived at the Fédération Wallonie-Bruxelles, he found a ministry trapped in a cycle of technical debt and operational inefficiency that many large organizations will recognize. FWB’s existing data architecture had evolved organically over years, creating a patchwork of disconnected systems that hindered rather than helped their mission to serve Belgian citizens effectively. Martin’s team found it difficult to track changes or ensure data quality, and they longed to empower users to build and maintain their own data workflows.

The critical issues that drove change include: 

• Closed-source tool dependency that limited flexibility and transparency in government operations
• Complete absence of version control for data pipelines, making changes risky and difficult to track
• Limited or non-existent data observability into pipeline health, performance, or failures
• Significant barriers to user onboarding that prevented teams from becoming self-sufficient with data

The ministry's technical stack reflected this complexity, with data flowing from SQLServer, MySQL, PostgreSQL, Denodo, DB2, FTP files, and various APIs into their Azure Data Lake and BI tools like PowerBI and Metabase. “Most data transformations had to be done manually or through Excel,” Martin says. “This was a nightmare because each year we needed to reprocess everything, and I have colleagues that had to spend two or three months making modifications to one data set.”

Choosing flexibility and modern engineering practices

After struggling with closed-source tools and fragmented data architecture, Martin and his team systematically evaluated their orchestration options to find a platform that could unify their complex ecosystem while granting more accessibility for analytics users.

The ministry tested multiple solutions including TiMi, KNIME, Apache Airflow, and Prefect before choosing Dagster. Their hands-on experience with Airflow for several weeks, followed by months with Prefect, gave them clear insights into what they needed to avoid. "Airflow was not intuitive for our users—the UI lacked user-friendliness, concepts were hard to grasp and its logging capabilities were limited," Martin explains.

After a thorough evaluation, Dagster emerged as the clear choice for several key reasons:

• Asset-centric architecture: Dagster's focus on data assets as first-class citizens aligned perfectly with how their teams conceptualized data workflows, rather than just thinking about tasks and processes.
• Modern developer experience: The platform offered clean development workflows, local testing capabilities, and software engineering best practices that their technical teams valued.
• Advanced observability: Built-in monitoring and debugging capabilities were essential for managing their complex, heterogeneous data environment across multiple sources and destinations.
• Organizational adaptability: Dagster's flexible architecture could accommodate FWB’s five “Administrations Générales” working across very different domains.
• Future-proof foundation: The platform could serve both their immediate orchestration needs and provide a foundation for advanced analytics and AI/ML capabilities.

Dagster's differentiated approach resonated immediately with their needs. The asset-centric architecture aligned perfectly with how their teams actually thought about data, while the modern developer experience promised to overcome the friction and inaccessibility that had historically prevented widespread adoption of data tools across the ministry. "Dagster was particularly well-suited to the polycephalous structure of our organization,” Martin says, referring to FWB’s many internal administrations, and noting that Dagster was the only solution they’d found that offered advanced data observability across such a divergent data estate.

We greatly appreciate that Dagster treats assets as first-class citizens in its architecture. This capability for comprehensive tracking and management of data assets throughout their lifecycle is crucial for us.

Building a developer-first data platform with Dagster

Rather than attempting a big-bang migration, the Fédération Wallonie-Bruxelles adopted a thoughtful, phased approach that prioritized developer experience and organizational change management — and lots of onboarding. “This is a ministry in the public domain, and the data literacy in our organization is not that good yet,” Martin says. “We knew adoption success will be only as good as our support for our users.”

FWB’s strategic migration approach included: 

• Phased rollout strategy to minimize disruption and ensure smooth transitions
• Hands-on user support with tailored onboarding sessions for each team
• Comprehensive training programs covering fundamentals like SQL, Python, Git, and Docker
• Best practices development to ensure consistency and quality across all pipeline development
• Custom Python package creation to simplify interactions with their unique ecosystem

The technical implementation showcased sophisticated engineering practices that maximized Dagster's capabilities while addressing the ministry's specific requirements:

• Custom Dagster resources and IO managers for SQLServer, PostgreSQL, DB2, and various APIs
• Seamless dbt integration using Dagster's built-in support for data transformation workflows
• DuckDB integration for efficient data processing and analysis
• Automated project integration using Git submodules and automated workflows
• GitLab CI/CD pipeline automation with testing, validation and deployment on Docker Swarm infrastructure – provided by our IT partner (ETNIC) – across feature branch, staging, and production environments

One of their most innovative implementations demonstrates the power of treating infrastructure as code: "We developed a workflow to automatically integrate dbt projects into our Dagster code locations. This workflow implies the separation of code repositories for dbt projects and Dagster code locations. On each dbt project release, we automatically include the dbt project as a Git submodule into the Dagster code location."

This approach enables their users to leverage existing dbt projects without requiring deep Dagster knowledge, significantly lowering the barrier to entry while maintaining the benefits of unified orchestration.

I really wanted our data analysts to feel confident and not intimidated — for them, even using a CLI could be challenging. With Dagster we could give them their own environment, included in the orchestration.

Measurable improvements in reliability and productivity

Six months after implementation, the Fédération Wallonie-Bruxelles has achieved transformational results that demonstrate Dagster's impact on both technical operations and business outcomes. The combination of improved reliability, enhanced developer productivity, and proactive data management has fundamentally changed how the ministry approaches data.

FWB’s quantifiable operational improvements include:

• Pipeline reliability enhancement through comprehensive monitoring and alerting via MSTeams and NTFY notifications
• Faster issue resolution enabled by Dagster's observability features for quick problem identification
• Improved data quality detection helping software developers identify and resolve application-level data issues
• Reduced operational disruptions through more reliable pipeline execution and monitoring
• Enhanced decision-making confidence enabling proactive proposal of new data products to users

The transformation extends beyond mere technical metrics to cultural and organizational changes that demonstrate how the broader value of modern data orchestration brings organizational and strategic benefits:

• Shift from reactive to proactive data product development and user support
• Increased data literacy across the organization through improved training and documentation
• Better stakeholder confidence in data quality and delivery timelines
• Enhanced collaboration between technical and non-technical teams
• Stronger foundation for future AI/ML initiatives and advanced analytics

The ministry's monitoring approach exemplifies how proper observability transforms operations: "We set up runs and data quality checks monitoring (with notifications) to ensure that our data pipelines are running as smooth and efficiently as possible,” Martin explains.

This is how, he says, the FWB data team went from discovering problems only after dashboards broke to preventing issues before they can impact end users — a crucial capability when serving 4.5 million citizens who depend on reliable government services.

Expanding capabilities and adding AI

The Fédération Wallonie-Bruxelles sees Dagster as the foundation for ambitious next steps that will further democratize data access and, eventually, leverage cutting-edge AI and ML technologies to better serve their citizens.

FWB’s immediate roadmap priorities include:

• AI-powered user assistance through MCP service integration connecting their custom Dagster library documentation to an LLM for personalized, context-aware help
• Enhanced onboarding exploring Dagster's new dg CLI to further streamline user adoption
• Advanced analytics capabilities building on their solid orchestration foundation to support growing AI/ML workloads

The ministry's long-term vision positions Dagster as the central component in a comprehensive data ecosystem that supports unified, efficient data services across all stakeholder needs while promoting data literacy and engineering best practices throughout the organization.

"We plan to integrate an MCP service that connects our custom Dagster library documentation and data discovery capabilities to an LLM, enabling personalized, context-aware assistance for our users," Martin explains. "Thanks to Dagster, dbt, DuckDB and our data lake, we are building the foundation of a data platform that can grow and flex to support the needs of our organization for a very long time to come."

‍