0

I saw a kafka warning for topics my current instance is not subscribed to. We use the same kafka broker, and each dev has his/her own topic. However we all use the same default groupID/clientid specified in a properties file.

2017-06-27T11:48:35.20-0400 [APP/PROC/WEB/0]OUT 2017-06-27 15:48:35,195 WARN [NetworkClient] - [kafka-coordinator-heartbeat-thread | ingestion-matching-kafka-consumer-group] - Error while fetching metadata with correlation id 154274 : {to-process-yi=UNKNOWN_TOPIC_OR_PARTITION, to-process-shawn=UNKNOWN_TOPIC_OR_PARTITION}
2017-06-27T11:48:35.34-0400 [APP/PROC/WEB/0]OUT 2017-06-27 15:48:35,339 INFO [PdfXmlMatcherTask] - [scheduler-1] -
2017-06-27T11:48:35.34-0400 [APP/PROC/WEB/0]OUT 2017-06-27 15:48:35,340 INFO [PdfXmlMatcherTask] - [scheduler-1] - Checking messages...
2017-06-27T11:48:35.41-0400 [APP/PROC/WEB/0]OUT 2017-06-27 15:48:35,412 WARN [NetworkClient] - [scheduler-1] - Error while fetching metadata with correlation id 154275 : {to-process-yi=UNKNOWN_TOPIC_OR_PARTITION, to-process-shawn=UNKNOWN_TOPIC_OR_PARTITION}

Is this expected? Do I need separate group/client IDs even though each person has their own topic?

Secondly I'm not quite sure what this error means: the heart-beat coordinator thread couldn't fetch metadata for 2 topics, but these topics exist, it's just that the consumers for those topics aren't running.

kyl
  • 475
  • 1
  • 5
  • 16

1 Answers1

1

client.id - is useful for tracing requests, should be unique to each client (though not mandatory).

groupid - this applies only to consumers. By your short description you probably want each consumer in its own group.

I suggest you read what ConsumerGroups are for : "Kafka scales topic consumption by distributing partitions among a consumer group, which is a set of consumers sharing a common group identifier. " https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/

UNKNOWN_TOPIC_OR_PARTITION is an error that the broker returns exactly when a topic or partition does not exist. it's an erro that the Kafka client treats as retriable.

If you think that the topic exists, please check twice. It could also be the case that the topic exists but a producer or consumer is targeting directly a non-existing partition of an existing topic.

Edoardo Comar
  • 531
  • 2
  • 5
  • It seems like the topics were deleted and recreated with a similar name. Since it was the same consumer group (even though it wasn't subscribed to those topics) I got error messages from the broker anyways? – kyl Jun 28 '17 at 14:17