The video on-demand of this session is available to logged in QCon attendees only. Please login to your QCon account to watch the session.

Session + Live Q&A

Building and Scaling a Control Plane for 1000s of Kafka Clusters

At Confluent Cloud we deploy, manage, maintain and scale 1000s of Kafka, Kafka Connect and KSQL clusters. In order to create a cloud-native experience for our users, we've built a control plane that provides both self-serve functionality for our users and automation for Confluent engineers. We ask for a lot from our control plane - it has to support all the services that are part of Confluent Platform, it has to give each service owner significant control over their service configuration and the ability to extend the functionality for their service independently, it has to abstract away the differences between the various clouds and network architectures. Most importantly - it has to scale on multiple dimensions such as number of clusters, API keys, high rate of configuration changes, etc.    

In order to support all these requirements, we've been practicing what we preach - our cloud control plane uses an event-driven architecture based on Apache Kafka. In this talk, we'll cover some of our architecture highlights, the interesting challenges we've encountered as we scaled our cloud usage, and how we evolved a control plane for Kafka while evolving Kafka for the cloud.


Gwen Shapira

Software Engineer @Confluent, PMC Member @Kafka, & Committer Apache Sqoop

Gwen is an engineering leader at Confluent, managing the Cloud-Native Kafka team. She has 15 years of experience working with code and customers to build scalable data architectures, integrating microservices, relational and big data technologies. She currently specializes in building real-time...

Read more
Find Gwen Shapira at:


Vivek Sharma

Senior Engineer @Confluent

Vivek is a Senior Software Engineer at Confluent where he applies 10+ years of experience with the world's largest distributed systems to the task of creating a fully self-serve control plane that orchestrates thousands of mission critical clusters. He is a fan of decoupled distributed...

Read more


Tuesday May 18 / 12:00PM EDT (40 minutes)


Architectures You've Always Wondered About


KafkaApacheArchitectureCloud NativeStreaming DataClustering & CachingClusters

Add to Calendar

Add to calendar


From the same track

Session + Live Q&A Architecture

BBC Online: Architecting for Scale with the Cloud and Serverless

Tuesday May 18 / 10:00AM EDT

During the 2020 US Presidential Election, BBC Online served video, audio, and text to over 140 million users. Events like this require the BBC’s sites and apps to be at their very best – fast, reliable, and relevant at massive scale. It’s been achieved with a modern,...

Matthew Clark

Head Of Architecture for the @BBC's Digital Products

Session + Live Q&A Performance

Optimizing Your Web Performance: Separating the Signals from the Noise

Tuesday May 18 / 11:00AM EDT

Everyone agrees on the theory of optimizing web performance but how does it work in practice? In this talk I will be sharing the journey Trainline has been on leading up to Google introducing Core Web Vitals as a ranking signal this month. It will include how as a team we learned to separate out...

Carl Anderson

Director of Engineering @trainline

PANEL DISCUSSION + Live Q&A Architecture

Panel: Event Driven Architectures of Scale

Tuesday May 18 / 01:00PM EDT

Event-driven architectures (EDA) have been around for decades but are back in vogue, driven in part by the shift to the cloud and multicore architectures, data-driven applications, and related trends such as microservices and distributed computing.  Our expert panel will explore what...

Wes Reisz

Technical Principal @thoughtworks & Creator/Co-host of #TheInfoQPodcast, previously Platform Architect @VMware

Matthew Clark

Head Of Architecture for the @BBC's Digital Products

Gwen Shapira

Software Engineer @Confluent, PMC Member @Kafka, & Committer Apache Sqoop

Ian Thomas

Senior Principal Engineer at Flutter Entertainment

View full Schedule