Redpanda: A new storage engine with Kernel bypass technologies for 10x lower tail latencies

“Redpanda: A new storage engine built from the ground up for Kafka-API compatibility with Kernel bypass technologies for 10x lower tail latencies” by Alexander Gallego
In this talk we’ll cover how we built a new storage engine from scratch using a thread per core architecture for predictable tail latencies. We’ll dive deep into the challenges of adapting a well known protocol (Kafka-API) on top of this new storage engine and the performance gains of optimizing for modern hardware.
Event page: performancesum...
Slack QA channel: performancesum...

[@mjwchapman]

this was brilliant. thank you very much. next stop ScyllaDB

[@gukki5]

he stated that intra-datacenter network latencies nowadays between machines was on par with inter-NUMA latencies. that’s just categorically untrue. 100s of nanoseconds vs 100s of microseconds. that’s 3 orders of magnitude.

[@alexandergallego5899]

i mentioned for a 'contended' resource. I've measured in the milliseconds.

[@alexandergallego5899]

but to add, you *can* in fact do a network call in low double digit microseconds. It's easy to measure, just hook up 2 servers back to back with some SPF link or smth and DPDK and you'll see. All of this is quite easily verifyable really.

[@kokizzu]

Why I never see throughput benchmark for this XD.

[@GlebWritesCode]

"Developers love Kafka api" - are you serious? After a year working with Kafka, I know some configuration choices and design problems I still shudder at

[@bruceritchie7613]

Oh hell yes. Completely agree.

[@alexandergallego5899]

lossy compression here. what ppl want is their existing apps to go faster w/ no code changes. the partitioning scheme of un ordered collection with totally ordered sub collections is pretty handy as a modeling. what you point out is the heavy weight nature of partitions which is true, but the mental model is helpful.

[@cepuofficial9025]

Any tutorial on this?

[@tarastsugrii4367]

vectorized.io has an extensive documentation for Redpanda - vectorized.io/docs.

[@pingkai]

Anybody really cares about latency what to use a RPC and streaming? If I can tolerate 18ms I did not see why I can not tolerate 100ms.

[@tarastsugrii4367]

well, spending $1m vs $10m on compute makes a good case for preferring one over the other ;)

[@Shogoeu]

What happened to Concord.io?

[@alexandergallego5899]

sold it to akamai.