I'm looking for some clarity as to if the Kafka Table engine supports exactly once semantics. I understand that clickhouse uses librdkafka, and that librdafka supports EOS as of v1.4. The latest versions of clickhouse are using librdkafka v1.5+. Is use of the library enough to confirm that EOS is supported by the kafka table engine, or does the table engine functionality require additional changes to support EOS? It isn't clear to me from the clickhouse documentation.
Asked
Active
Viewed 459 times
1 Answers
1
No EO with Clickhouse until two-phase-commit is implemented in CH.
https://github.com/ClickHouse/ClickHouse/issues/18668#issuecomment-752946654
BTW - while EOS semantics will guarantee you that no duplicates will happen on the Kafka side (i.e. even if you produce the same messages few times it will be consumed once), but ClickHouse as a kafka client can guarantee only at-least-once. And in some corner cases (connection lost etc) you can get duplicates. We need to have something like transactions on clickhouse side to be able to avoid that.
Denny Crane
- 11,574
- 2
- 19
- 30