3

One of my services stopped working a few hours ago. I have restarted the server but this consumer process just isn't registering with Kafka. This is what shows up when the consumer is started :

2017-04-21 10:22:54.887  INFO 18036 --- [afka-consumer-1] o.a.k.c.c.internals.ConsumerCoordinator  : Revoking previously assigned partitions [] for group mysql-conversions-group
2017-04-21 10:22:54.890  INFO 18036 --- [afka-consumer-1] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked:[]
2017-04-21 10:22:54.890  INFO 18036 --- [afka-consumer-1] o.a.k.c.c.internals.AbstractCoordinator  : (Re-)joining group mysql-conversions-group
2017-04-21 10:22:54.915  INFO 18036 --- [afka-consumer-1] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator li819-138.members.linode.com:9092 (id: 2147483646 rack: null) dead for group mysql-conversions-group

I have 4 different consumer groups and they are running just fine. Any thoughts?

UPDATE

Its odd really. We had to restart another one of our Consumer, and after restart, the other consumer that was working fine a second ago is now logging the same error. Basically, we now have 2 consumer processes that can't connect to Kafka for some reason.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Ankush92
  • 401
  • 1
  • 9
  • 20

1 Answers1

3

Answering my own question:

Finally got it to work. This was not a broker issue or a consumer issue. What happened was, we had enabled firewall on both of our Kafka brokers around 8:30 in the night. The consumers that were already connected with Kafka were working fine. However, when one of these consumers were restarted, they were unable to connect to the coordinator node. We disabled firewall on both the Kafka brokers, and the data was immediately relayed to the consumers. It is a temporary fix ATM, but at least we have something.

Ankush92
  • 401
  • 1
  • 9
  • 20