Building and Scaling a Control Plane for 1000s of Kafka Clusters

At Confluent Cloud we deploy, manage, maintain and scale 1000s of Kafka, Kafka Connect and KSQL clusters. In order to create a cloud-native experience for our users, we've built a control plane that provides both self-serve functionality for our users and automation for Confluent engineers. We ask for a lot from our control plane - it has to support all the services that are part of Confluent Platform, it has to give each service owner significant control over their service configuration and the ability to extend the functionality for their service independently, it has to abstract away the differences between the various clouds and network architectures. Most importantly - it has to scale on multiple dimensions such as number of clusters, API keys, high rate of configuration changes, etc.    

In order to support all these requirements, we've been practicing what we preach - our cloud control plane uses an event-driven architecture based on Apache Kafka. In this talk, we'll cover some of our architecture highlights, the interesting challenges we've encountered as we scaled our cloud usage, and how we evolved a control plane for Kafka while evolving Kafka for the cloud.