Strange Loop

EoS in Kafka: Listen up, I will only say this once!

Apache Kafka's rise in popularity as a streaming platform has demanded a revisit of its traditional at-least-once message delivery semantics. In this talk, we present the recent additions to Kafka to achieve exactly-once semantics (EoS) including support for idempotence and transactions in the Kafka clients. The main focus will be the specific semantics that Kafka distributed transactions enable and the underlying mechanics which allow them to scale efficiently. We will discuss Kafka's spin on standard two-phase commit protocols, how transaction state is maintained and replicated, and how different failure scenarios are handled. We will also share our view of future improvements that will make exactly-once stream processing with Kafka even better!

Jason Gustafson

Confluent

I am an engineer at Confluent and an Apache Kafka committer. I am currently one of the top contributors to Kafka and part of the team that designed and implemented support for exactly-once processing in Kafka.