Stephan Ewen - Fixin the Hard Bits of Event Processing with Restate & Kafka

Event-driven applications are still hard in 2024, despite all their strong and desirable properties (scalability, resilience, ability to replay, ...). While streaming analytics are supported by a wave of powerful tools (like Flink, KStreams, Materialize, RisingWave, ...), the transactional side of event-driven applications is not quite as far.
Complex per-event control flow, complicated state semantics, and interactions with many external dependencies (APIs/services/...) make those applications challenging. Observability and debuggability are still ways behind what RPC-based architectures offer with today's tools.
In this talk we present 'Restate' (restate.dev/), a new open source tool we have been working on. Restate complements systems like Kafka when building event-driven workflows, async tasks, distributed signaling, or even full microservices architectures. While Restate internally is an event-driven system (sharing some similarities with a Kafka/Flink/KStreams crossover), it has a very simple programming model: Durable Execution (for event-handlers and RPC), Durable Promises (for signaling), and Virtual Objects. Those three constructs are the resilient encapsulations of the core ingredients of every distributed application: compute, communication, and state.
In the talk, we will first we'll look practically at common challenging patterns: Where do those tend to break current architectures? How can we address those issues with Kafka/Restate?
- burst-y loads / rapid scaling
- head-of-the-line waiting on long event processing times
- combining Kafka events with webhooks and other signals
- Transactionally updating databases
- high-fan-out routing to consumer APIs
- state machines with long-running transitions
- delaying/postponing certain events
Second, we take a look at similarities, synergies, and differences between Restate, Kafka, and Stream-Processing. We look at the way applications are expressed, how the systems organize event logs and event processing, and how this mirrors the differences between analytical- and transactional applications respectively.
Speaker: Stephan Ewen
More: 2024.berlinbuz...
###
Follow us on Social Media and join the Community!
Mastodon: floss.social/@...
LinkedIn: / berlin-buzzwords
Instagram: / berlinbuzzwords
Website: 2024.berlinbuz...
Berlin Buzzwords is an event by Plain Schwarz - plainschwarz.com