Notification Service With Kafka In King
Description
Form popularity
FAQ
In order to guarantee the exactly-once semantics a consumer must be configured with read isolation. level of READ_COMMITTED. This ensures it will not read transactional messages written to topic partitions until the message is marked as committed.
Kafka can not push notifications to consumers. Consumers have to keep polling/consuming messages in order to receive them. This is the only way messages stored in Kafka can flow to consumers. The Consumer Design section in the Apache Kafka documentation explains why "consumer pulling" was choosed over "broker pushing".
Kafka supports exactly-once delivery in Kafka Streams and uses transactional producers and consumers to provide exactly-once delivery when transferring and processing data between Kafka topics.
Kafka stands out for its ability to handle high throughput and provide scalability, fault tolerance, and message retention. These features make Kafka ideal for a notification system where reliability and real-time processing are crucial.
The first step in achieving Exactly-Once processing is to enable idempotence by setting enable. idempotence=true in the producer config. This will enable Idempotence Guarantee Semantics. We can optionally set transactional.id to some unique value that is constant across application restarts.
One very important and often missed detail is that Kafka supports exactly-once delivery only in Kafka Streams. To turn it on just change a config option processing. guarantee from at_least_once(default option) to exactly_once_v2.
The notification email service consumes email requests from the Kafka notification topic and processes them to send them to a third-party service. Modules like PT, TL, PGR etc make use of this service to send messages through the Kafka Queue.
While Kafka has supported exactly-once semantics for years in its clients and stream processing libraries, Kafka Connect lacked this support for source connectors until version 3.3. In this talk, learn how KIP-618 made exactly-once source connectors possible.
