0

I am using Redpanda with Flink for streaming messages and processing them with multiple jobs each reading from the same topic and in same group, I am giving group id while consuming data messages using KafkaSource but it does not seem to be taking it.

I tried enabling checkpointing and setting commit.offsets.on.checkpoint to true according to docs but the result was same

KafkaSource<Map<String, Object>> logSource = KafkaSource.<Map<String, Object>>builder()
                .setBootstrapServers(BOOTSTRAP_SERVER)
                .setTopics("logs")
                .setProperty("group.id", "group1")
                .setProperty("commit.offsets.on.checkpoint", "true")
                .setStartingOffsets(OffsetsInitializer.latest())
                .setValueOnlyDeserializer(new LogDeserializer())
                .build();

Also if two consumer in the same group consuming from same topic each consumer should consume unique records, for e.g. if I send 10 events in topic X and the two consumer are in same group consuming from that topic then each should not receive same event, But that is not happening in this case

Krutik Gadhiya
  • 1
  • 1

2 Answers2

0

If you are submitting multiple Flink jobs that are reading from the same topic, each job will start reading according to the strategy that you're defining with setStartingOffsets.

Also if two consumer in the same group consuming from same topic each consumer should consume unique records, for e.g. if I send 10 events in topic X and the two consumer are in same group consuming from that topic then each should not receive same event, But that is not happening in this case

Flink only uses the consumer group in order to expose the progress of consumer and consuming group for monitoring. It does not use the consumer groups for anything else.

Martijn Visser
  • 1,468
  • 1
  • 3
  • 9
0

In the first case, did you receive any records or not?

Dunith Dhanushka
  • 4,139
  • 6
  • 26
  • 29