FlixBus: Data Mesh for 50+ Teams

Executive Summary

Flix scales Kafka data-streaming operations from 25 to 50 teams with Conduktor—optimizing fleet tracking, route efficiency, and AI-driven customer experiences across Europe and North America.

Kafka at Flix

Operating at the crossroads of a tech startup, e-commerce platform, and transportation company, data is the lifeblood of Flix. Kafka plays a critical role in managing vast streams of real-time data, from vehicle locations to customer feedback, enabling Flix to:

• Dynamically modify bus schedules on-the-fly
• Optimize routes by analyzing traffic patterns and vehicle data
• Rapidly respond to customer feedback, enhancing service quality

Challenge

As Flix expanded, so did the complexity of its Kafka infrastructure. Within five years, Kafka utilization expanded to over 2,300 topics and 9,500 partitions across 10 domains and 50 teams.

Managing a data operation of this size presented multiple challenges:

• Complex permissions — Managing access controls across global teams while adhering to local data governance and privacy regulations
• Access bottlenecks — No single interface for technical and non-technical users to independently access Kafka data
• Limited discoverability — Analysts struggled to discover available data, view schemas, and filter streams without a platform for observability
• Cost management — Without a unified view, proactively consolidating data pipelines to prevent runaway costs was challenging

Solution

At Flix, robust data security is paramount. The sensitive nature of payment transactions and personal information required for travel bookings made data protection a critical concern.

Flix adopted a resource-centric approach grounded in data mesh principles, giving teams ownership of their data within a federated control framework.

Data mesh approach

• Resource-focused permissions — Permissions tied to resources, centralizing control over crucial assets
• Distributed governance — Resource owners manage access to their respective resources
• Centralized framework — All oversight remains under a cohesive framework ensuring consistency and compliance

Modular data architecture with Conduktor

Flix implemented a modular architecture using individual YAML files for each team's Kafka resources:

• Clear ownership — Clearly identifies the team responsible for resources within a namespace
• Contained leakage — Limits problems with overly lenient access settings to affected resources
• Auditable changes — CI/CD tools track all changes with merge history
• Promotes collaboration — Teams request access through pull requests; owners approve without centralized authority

How it works

Flix's federated Kafka security management operates through open-source technologies and structured CI/CD workflows:

1. Each team has a Kafka resource-oriented YAML file defining the owner and permissions granted to other teams
2. Changes trigger CI/CD workflows that validate configurations through federated governance
3. Files are transformed into a format compatible with Kafka Security Manager (KSM), a Conduktor open-source project
4. The same files propagate permissions into Conduktor for interaction and troubleshooting

This workflow ensures each team is responsible for their resources while permissions are uniformly applied:

• Kafka ACLs — Control application access, ensuring only authorized applications interact with specific topics
• Conduktor RBAC — Provide team members with permissions to manage, monitor, and troubleshoot their Kafka resources

Results

Through collaboration with Flix, Conduktor developed a self-service framework enhancing collaboration between Platform and Application teams:

• 50+ teams working independently with Kafka
• 2,300+ topics managed across 10 domains
• Reduced reliance on central teams with intuitive self-service controls
• Automated compliance with configurable approval workflows
• Enforced best practices on Kafka configurations, maintaining consistency

Frequently Asked Questions

What is a data mesh approach to Kafka governance?

Data mesh is a decentralized approach where teams own their data within a federated control framework. Flix uses resource-focused permissions with distributed governance while maintaining centralized oversight for consistency.

How does Flix manage Kafka permissions across 50+ teams?

Flix uses individual YAML files for each team's Kafka resources, tracked in CI/CD with merge history. Teams request access through pull requests that resource owners approve without centralized authority.

What is Kafka Security Manager (KSM)?

KSM is a Conduktor open-source project that transforms YAML configurations into Kafka ACLs. Flix uses it alongside Conduktor RBAC to control both application and team access.