0

So, i faced interesting situation:

In my integration test i send data to 2 topics, and they should be consumed by application by the order. At first - userTopic, then shoud be consumed userOrderTopic.

When i send my messages in test, it sends really rapidly, at the same time. But sometimes, the order of consuming by application is different. (userOrder topic, then userTopic) what breaks my test at all.

I found some triks, how i could avoid it - thread sleep, etc. But i believe that it's bad solution.

I am new with kafka, as result with KafkaTestUtils. Is there any method, which could check that message in topic was consumed/check until that message was consumed?

Something like this:

//check that message in userTopic was consumed, to have a chance send message to userOrderTopic 
KafkaTestUtils.waitUntilBeConsumed(serverBootstrap, "user-consumer", topic, 0, messageToUserTopic)

or

KafkaTestUtils.getNowConsumedMessage(serverBootstrap, "user-consumer", topic, 0)

P.S. in my intergration test i don't have accsess to producer/consumer.

Or maybe in spring.kafka /spring.kafka.test are another tools to do it in proper way?

  • I don't think the design is correct: if you want to consume those records in the order, then they have to go to the same topic and same partition. We cannot control how `KafkaConsumer` does its subscription and polling when we provide several topics. This is really not how messaging is work: two different sources must be treat as independent and the order consuming from them should not be taken into the consideration. – Artem Bilan Nov 23 '21 at 21:03
  • @ArtemBilan but, in case with same topics we could have same situation, when order would be consumed before user, right? Basically, they are not independent, because we need to have some information about user, when we would get user order. In real microservice flow it is impossible, to get wrong flow, but in test it is real problem ) –  Nov 23 '21 at 21:11
  • If data placed into the same partition, then it is consumed in the order it was added there. – Artem Bilan Nov 23 '21 at 21:16

1 Answers1

0

KafkaTestUtils is only for consuming from test consumers not real application consumers.

There is no easy solution for this race condition; you need to wait for the first record to be consumed before sending the second.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • So, there any way to check that message was cosumed? Than looks like thread.sleep is not bad idea –  Nov 23 '21 at 21:05
  • There are a few different techniques, for example - if your listener calls a service, add a mock/stub of that service in your test case and wait for it to be called; - add a `RecordInterceptor` to the listener container factory in your test case and wait for the interceptor to be called; - add a proxy around the listener that counts down a latch in the test case - see https://stackoverflow.com/questions/53678801/kafka-consumer-producer-test-in-spring-kafka/53684073#53684073 for an example. – Gary Russell Nov 23 '21 at 21:19
  • thanks for explanation –  Nov 23 '21 at 21:25