Learnings of Running Kafka Tiered Storage at Scale

KIP-405 introduced tiered storage in Apache Kafka. It introduces the separation of compute and storage in brokers that improves the scalability, efficiency, and elasticity of Kafka clusters. We implemented this feature and have been running it in production for several months in different tiers of clusters at Uber.
We will talk about the following:
The principles followed in building the feature.
The journey of deploying and running it in our production clusters with different workloads.
The learnings from running it in production at a large scale, that led to a few interesting features extended from KIP-405.
The issues encountered and how we have fixed and mitigated them.
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion - designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#confluent #apachekafka #kafka