What Kafka Users Actually Asked Us at Kafka Summit London 2023
Field-level encryption, multi-tenancy, topic-as-a-service, and RBAC dominated conversations at our booth. Here's what the community cares about most.
One week after Kafka Summit London. We didn't attend a single talk. Our booth was under constant siege by visitors, and we spent two days in non-stop conversations.
Tom did manage to run his session on Safeguarding your Kafka Infrastructure. Worth a watch.
Here's what sparked the strongest reactions from the community.
Field-Level Encryption Without Code Changes
This feature captured the most interest by far.
Field-level encryption solves a concrete problem: GDPR, PII, HIPAA compliance. Everyone recognizes the need, few have good solutions.
Our approach: encrypt sensitive data with zero code changes. No modifications to your applications regardless of programming language, version, or OS. The data flows through Gateway and gets encrypted transparently.
Documentation on our Gateway Marketplace.
Multi-Tenancy for Cost Reduction
Multi-tenancy received overwhelming praise. We're fans too.
Think of it like the cloud movement in the 2000s: virtual machines gave you flexibility, scale, control, and isolation. We provide the same for Kafka clusters: Virtual Clusters.
Visitors immediately grasped the cost reduction potential.
Our Conduktor Free Playground runs over 4,000 Virtual Clusters on a single Confluent Cloud instance. That number on one instance without performance issues surprised everyone.
Documentation on our Gateway Marketplace.
Topic-as-a-Service Removes Human Bottlenecks
Data owners can now grant permissions directly to users without waiting for Platform Ops.
We've all been there: requesting access on Slack, explaining which team you belong to, explaining that your team lead is on holiday and you need access urgently, explaining that your team has a new name and everything needs updating.
Humans in the middle of any process means indefinite wait times.
Giving autonomy back to developers aligns with data ownership principles. It eliminates bottlenecks and creates a more efficient environment.
Documentation about Topic as a Service.
RBAC That Actually Scales
ACLs exist, but they don't scale. Managing ACLs across a large organization is cumbersome. Kafka should have provided RBAC natively.
Conduktor RBAC works for both users and applications:
- Users: access all topics because you're part of
@delivery - Applications: service accounts with scoped topic access
Combined with our Audit feature, you can track who accessed what, when, and how. Granular visibility for compliance and accountability.
Documentation about security features and RBAC on our Gateway Marketplace.
Virtual Topics Simplify Architecture
Our "SQL filter" feature creates Virtual Topics. Think PostgreSQL VIEWs, but for Kafka topics.
This removes moving parts from your architecture, accelerates delivery, and reduces costs. One of Conduktor's most underrated capabilities.
Everything Demonstrated Live
Nothing was theoretical. Every feature we discussed ran in a live demo.
The Gateway journey made sense: multi-tenancy, policies, virtual topics, data masking, field-level encryption, cold storage. Each feature builds on the previous.
What We're Building Next
We openly discussed our roadmap. Merging multiple clusters as one caught immediate attention.
What this enables:
- Data Mesh across multiple Kafka clusters
- Automatic failover without application changes
- No more replicator or mirror-maker to duplicate data
- With caching, effectively a CDN for Kafka (Kafka Cache Documentation)
Thank You
Thanks to everyone who stopped by. The conversations were valuable, and we enjoyed answering complex questions about what's under the hood.
Next stop: Current 2023 in California.