3

I have a Java consumer that runs continuously with a lot of data and messages.. sometimes, the program gets stuck on consumer.poll the data.. it seems like deadlock or infinite loop but I’m not sure about it. There is no exception thrown and the program seems to work properly, although it doesn’t polls the actual data. (The problem happens only in the java program.. I run console consumer and it works great)

The only way to fix it, is by restarting the program...

public class Main {
Properties consumerComfig = new Properties ();
consumerConfig.put(ConsumerConfig. KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArrayDeserializer");
consumerConfig.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,"Org.apache.kafka.comnect.json.JsonDeserializer");
consumerConfig.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "some server")
consumerConfig.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID());
consumerConfig.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"):
consumerConfig.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 30000);
consumerConfig.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG,10000)
consumerConfig.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest"):
consumer = new KafkaConsumer<String, JsonNode> (consumerConfig);

public void consumeAndSendToCorrelator () {
    logger.info ("Starting kafka consumer, listening to topics stringBuilder.toString ()");
    try {
        consumer.subscribe("INPUT TOPICS");
        while (true) {
            ConsumerRecords<String, JsonNode> records = consumer.poll(8000);
            logger.debug("Trying to pull records from consumer");
            JsonNode value = null;
            for (Consumer Record<String, JsonNode > irecord :records){
                try {
                    value = record.value();
                    if (value == null)
                        continue;
                    //Somme thread that process the message
                    correlatorActor.tell(value, ActorRef.noSender);
                } catch (JsonProcessingException e) {
                    logger.error("Could not parse incoming json" + value.toString());
                } catch (IOException e) {
                    logger.error("IOException occurred", e);
                } catch (Exception e) {
                    logger.error("Error occurred", e);
                }
                consumer.commitSync();
            }
        }
    }catch(WakeupException e) {
        logger.info("closed consumer");
    }finally {
        consumer.close()
    }
}

}

I use Kafka 0.10 Do you know what happens to it ? What can I do to fix it?

ronenpi18
  • 56
  • 4
  • wouldn't that depend on what your consumer does, and how is it configured? – eis Jul 02 '19 at 07:33
  • Given that plenty of people do not have this problem, it's most likely something in your setup or your code. It very likely helpful to post an [mcve] that exhibits the problem on your system. Also important to mention what version is your Kafka server and what version is your client library. – Erwin Bolwidt Jul 02 '19 at 08:03
  • Hey, I have configured it on the most basic way. And my java api is 0.10 too – ronenpi18 Jul 02 '19 at 08:13
  • @ErwinBolwidt hey, I added a code – ronenpi18 Jul 02 '19 at 12:43
  • How do you know that it gets stuck at `poll` and not `commitSync`? Also, why do you commit after every single processed message? You will commit the same position (determined by what was fetched in the `poll`-call) every time. – KWer Jul 03 '19 at 10:42
  • @KWer hey, I debugged the app few times and it fails on the poll function .. not on the commit.. I once used a commitAsync but it isn’t the case and I got problem with it.. can it be anything of configurations? What else you suggest me to try? – ronenpi18 Jul 06 '19 at 20:41
  • Honestly, I'm not too familiar with 0.10. Is it 0.10.0 or 0.10.1? The difference between the two is that for the latter, the heartbeats are sent from a separate thread. What do you see in the logs once the consumer gets stuck? Is there anything in the broker logs? At least after `max.poll.interval.ms` has passed (the default being 30s), the consumer should leave the group. – KWer Jul 09 '19 at 08:40

0 Answers0