Spring Kafka Unit Tests Triggers the listener, but the method cannot get the message using consumer.poll

Question

We are using spring-kafka-test-2.2.8-RELEASE. When I use the template to send the message, it triggers the listener correctly, but I can't get the message content in the consumer.poll. If i instantiate the KafkaTemplate without "wiring" it in a class attribute and Instantiate it based on a producer factory, it sends the message, but does not trigger the @KafkaListener, only work if I setup a Message Listener inside the @Test Method. I need to trigger the kafka listener and realize which Topic will be called next("sucess" topic when executed without errors, and "errorTopic" the listener throws an Exception) and the message content.

    @RunWith(SpringRunner.class)
    @SpringBootTest
    @EmbeddedKafka(partitions = 1, topics = { "tp-in-gco-mao-notasfiscais" })
    public class InvoicingServiceTest {

         @Autowired
         private NFKafkaListener nfKafkaListener;

         @ClassRule
         public static EmbeddedKafkaRule broker = new EmbeddedKafkaRule(1, false, "tp-in-gco-mao- 
         notasfiscais");

         @Value("${" + EmbeddedKafkaBroker.SPRING_EMBEDDED_KAFKA_BROKERS + "}")
         private String brokerAddresses;

         @Autowired
         private KafkaTemplate<Object, Object> template;

         @BeforeClass
         public static void setup() {
                System.setProperty(EmbeddedKafkaBroker.BROKER_LIST_PROPERTY,
              "spring.kafka.bootstrap-servers");
         }

         @Test
         public void testTemplate() throws Exception {
               NFServiceTest nfServiceTest = spy(new NFServiceTest());

               nfKafkaListener.setNfServiceClient(nfServiceTest);
               Map<String, Object> consumerProps =  KafkaTestUtils.consumerProps("teste9", "false", broker.getEmbeddedKafka());
               consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
               consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
               consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, InvoiceDeserializer.class);
               consumerProps.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

               DefaultKafkaConsumerFactory<Integer, Object> cf = new DefaultKafkaConsumerFactory<Integer, Object>(
            consumerProps);

           Consumer<Integer, Object> consumer = cf.createConsumer();

           broker.getEmbeddedKafka().consumeFromAnEmbeddedTopic(consumer, "tp-in-gco-mao-notasfiscais");

           ZfifNfMao zf = new ZfifNfMao();
           zf.setItItensnf(new Zfietb011());

           Zfietb011 zfietb011 = new Zfietb011();
           Zfie011 zfie011 = new Zfie011();
           zfie011.setMatkl("TESTE");
           zfietb011.getItem().add(zfie011);
           zf.setItItensnf(zfietb011);

           template.send("tp-in-gco-mao-notasfiscais", zf);
           List<ConsumerRecord<Integer, Object>> received = new ArrayList<>();
           int n = 0;
           while (received.size() < 1 && n++ < 10) {
                ConsumerRecords<Integer, Object> records1 = consumer.poll(Duration.ofSeconds(10));
                //records1  is always empty
                if (!records1.isEmpty()) {
                    records1.forEach(rec -> received.add(rec));
                }
           }

           assertThat(received).extracting(rec -> {
               ZfifNfMao zfifNfMaoRdesponse = (ZfifNfMao) rec.value();
               return zfifNfMaoRdesponse.getItItensnf().getItem().get(0).getMatkl();
            }).contains("TESTE");
            broker.getEmbeddedKafka().getKafkaServers().forEach(b -> b.shutdown());
            broker.getEmbeddedKafka().getKafkaServers().forEach(b -> b.awaitShutdown());
            consumer.close();
        }

        public static class NFServiceTest implements INFServiceClient {
            CountDownLatch latch = new CountDownLatch(1);

            @Override
            public ZfifNfMaoResponse enviarSap(ZfifNfMao zfifNfMao) {
                ZfifNfMaoResponse zfifNfMaoResponse = new ZfifNfMaoResponse();
                zfifNfMaoResponse.setItItensnf(new Zfietb011());

                Zfietb011 zfietb011 = new Zfietb011();
                Zfie011 zfie011 = new Zfie011();
                zfie011.setMatkl("TESTE");
                zfietb011.getItem().add(zfie011);
                zfifNfMaoResponse.setItItensnf(zfietb011);
                return zfifNfMaoResponse;
            }
        }
    }

score 0 · Answer 1 · answered Feb 18 '20 at 14:47

0

You have two brokers; one created by @EmbeddedKafka and one created by the @ClassRule.

Use one or the other; preferably the @EmbeddedKafka and simply @Autowired the broker instance.

I am guessing the consumers are listening to different brokers; you can confirm that by looking at the INFO logs put out by the consumer config.

answered Feb 18 '20 at 14:47

Gary Russell

166,535
14
146
179

I've followed your advice but it keeps triggering the listener, but consumer.poll does not capture the topic content. – Lucas Lopes Feb 18 '20 at 17:38
Turn on DEBUG logging for `org.springframework.kafka`. If you can't figure it out from that, post the log someplace (e.g. GitHub Gist, pastebin, or similar. – Gary Russell Feb 18 '20 at 17:42
Pushed on github . https://github.com/lucaslopescunha/logspringkafka/blob/master/log.log – Lucas Lopes Feb 18 '20 at 18:22
Your producer and consumer are not connecting to the embedded broker. `bootstrap.servers = [RVMTDV1147.riachuelo.net:9092, RVMTDV1148.riachuelo.net:9092, RVMTDV1154.riachuelo.net:9092]`. With version 2.3 and later you can add `bootstrapServersProperty="spring.kafka.bootstrap-servers"` to `@EmbeddedKafka`. With earlier versions, add `@TestPropertySource(properties = "spring.kafka.bootstrap-servers = ${spring.embedded.kafka.brokers}")` to the test case. – Gary Russell Feb 18 '20 at 18:59
This was shown in [my answer to your earlier question](https://stackoverflow.com/questions/60197555/junit-tests-with-spring-boot-2-1-12-and-spring-for-apache-kafka-2-2-12). – Gary Russell Feb 18 '20 at 19:06
It is not strange that in line 685(for ConsumerConfig values: ) we see bootstrap.servers = [127.0.0.1:55769] , but in line 819(for ProducerConfig values: ) I see in the log bootstrap.servers = [RVMTDV1147.riachuelo.net:9092, RVMTDV1148.riachuelo.net:9092, RVMTDV1154.riachuelo.net:9092] ? – Lucas Lopes Feb 18 '20 at 20:35
No; it's not strange at all; the first one (the real consumer) is using the Boot configuration properties; the second one is yours which you explicitly set the bootstrap servers in your test (via the `KTU.consumerProps()`). The `@TestPropertySource` fixes the boot configuration by replacing the boot property with the address of the embedded broker. – Gary Russell Feb 18 '20 at 20:42

score 0 · Answer 2 · answered Feb 18 '20 at 16:21

I've followed your advice but it keeps triggering the listener, but consumer.poll does not capture the topic content.

@RunWith(SpringRunner.class)
@SpringBootTest
@EmbeddedKafka(partitions = 1, topics = { "tp-in-gco-mao-notasfiscais" })
public class InvoicingServiceTest {

     @Autowired
     private NFKafkaListener nfKafkaListener;

 @Autowired
 public EmbeddedKafkaBroker broker;

 @Autowired
 private KafkaTemplate<Object, Object> template;

 @BeforeClass
 public static void setup() {
        System.setProperty(EmbeddedKafkaBroker.BROKER_LIST_PROPERTY,
      "spring.kafka.bootstrap-servers");
 }

 @Test
 public void testTemplate() throws Exception {
       NFServiceTest nfServiceTest = spy(new NFServiceTest());

       nfKafkaListener.setNfServiceClient(nfServiceTest);
       Map<String, Object> consumerProps =  KafkaTestUtils.consumerProps("teste9", "false", broker);
       consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
       consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
       consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, InvoiceDeserializer.class);

       DefaultKafkaConsumerFactory<Integer, Object> cf = new DefaultKafkaConsumerFactory<Integer, Object>(
    consumerProps);

   Consumer<Integer, Object> consumer = cf.createConsumer();

   broker.consumeFromAnEmbeddedTopic(consumer, "tp-in-gco-mao-notasfiscais");

   ZfifNfMao zf = new ZfifNfMao();
   zf.setItItensnf(new Zfietb011());

   Zfietb011 zfietb011 = new Zfietb011();
   Zfie011 zfie011 = new Zfie011();
   zfie011.setMatkl("TESTE");
   zfietb011.getItem().add(zfie011);
   zf.setItItensnf(zfietb011);

   template.send("tp-in-gco-mao-notasfiscais", zf);
   List<ConsumerRecord<Integer, Object>> received = new ArrayList<>();
   int n = 0;
   while (received.size() < 1 && n++ < 10) {
        ConsumerRecords<Integer, Object> records1 = consumer.poll(Duration.ofSeconds(10));
        //records1  is always empty
        if (!records1.isEmpty()) {
            records1.forEach(rec -> received.add(rec));
        }
   }

   assertThat(received).extracting(rec -> {
       ZfifNfMao zfifNfMaoRdesponse = (ZfifNfMao) rec.value();
       return zfifNfMaoRdesponse.getItItensnf().getItem().get(0).getMatkl();
    }).contains("TESTE");
    broker.getKafkaServers().forEach(b -> b.shutdown());
    broker.getKafkaServers().forEach(b -> b.awaitShutdown());
    consumer.close();
}

public static class NFServiceTest implements INFServiceClient {
    CountDownLatch latch = new CountDownLatch(1);

    @Override
    public ZfifNfMaoResponse enviarSap(ZfifNfMao zfifNfMao) {
        ZfifNfMaoResponse zfifNfMaoResponse = new ZfifNfMaoResponse();
        zfifNfMaoResponse.setItItensnf(new Zfietb011());

        Zfietb011 zfietb011 = new Zfietb011();
        Zfie011 zfie011 = new Zfie011();
        zfie011.setMatkl("TESTE");
        zfietb011.getItem().add(zfie011);
        zfifNfMaoResponse.setItItensnf(zfietb011);
        return zfifNfMaoResponse;
    }
}
}

Don't add another answer - it doesn't answer the question; edit the question to show the new information instead. — Gary Russell, Feb 18 '20 at 18:46
I am learning. I thought I couldn't delete/edit. I didn't even know how to post codes... Next time I won't make this mistake again. And by the way, I've posted on github the log in the other question"https://github.com/lucaslopescunha/logspringkafka/blob/master/log.log — Lucas Lopes, Feb 18 '20 at 18:52

Spring Kafka Unit Tests Triggers the listener, but the method cannot get the message using consumer.poll

2 Answers2