2

I am using spring-boot-starter-parent version 1.5.0.RELEASE, spring-kafka version 1.0.0.RELEASE and spring-kafka-test version 1.0.0.RELEASE in an application which consumes messages from a Kakfa 0.9 cluster. I have a unit test for my consumer which used KafkaEmbedded but it fails since the broker port is picked up randomly. Is there a way I can set this broker property without changing versions? Or which versions should I use so as to not break anything?

Here is the code for the KafkaListener and KafkaConsumerTest.

Listener.java

@Service
public class Listener {

    private static final Logger logger = LoggerFactory.getLogger(Listener.class);
    private CountDownLatch latch = new CountDownLatch(1);

    @KafkaListener(topics = "topic", group = "group", containerFactory = "kafkaListenerContainerFactory")
    public void consumeClicks(@Payload String msg, @Header(KafkaHeaders.RECEIVED_PARTITION_ID) Integer partition, @Header(KafkaHeaders.OFFSET) Integer offset, Acknowledgment ack) throws Exception {
        logger.info(msg);
        latch.countDown();
        ack.acknowledge();
    }

    public CountDownLatch getLatch() {
        return latch;
    }
}

KafkaConsumerTest.java (EDIT)

@DirtiesContext
@SpringBootTest(classes = {SpringApplication.class})
@RunWith(SpringRunner.class)
public class KafkaConsumerTest {
    private static final Logger logger = LoggerFactory.getLogger(KafkaConsumerTest.class);
    private static String TEST_TOPIC = "topic";

    @ClassRule
    public static KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, true, TEST_TOPIC);

    public KafkaTemplate<String, String> template;

    @Autowired
    private KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;

    @Autowired
    private Listener listener;

    @Before
    public void init(){
        System.setProperty("spring.kafka.bootstrap-servers", embeddedKafka.getBrokersAsString());
        Map<String, Object> senderProps = KafkaTestUtils.senderProps(embeddedKafka.getBrokersAsString());
        senderProps.put("key.serializer", StringSerializer.class);
        ProducerFactory<String, String> producerFactory = new DefaultKafkaProducerFactory<String, String>(senderProps);
        template = new KafkaTemplate<>(producerFactory);
        template.setDefaultTopic(TEST_TOPIC);
    }

    @Test
    public void testConsume() throws Exception {
        String record = "message";
        template.sendDefault(TEST_TOPIC, record);
        logger.debug("test-consume sent record {}", record);
        listener.getLatch().await(1000, TimeUnit.MILLISECONDS);
        Assert.assertEquals(listener.getLatch().getCount(), 0);
    }
}
Community
  • 1
  • 1
AlphaEpsilon
  • 141
  • 3
  • 11

2 Answers2

0

Please use spring-kafka 1.3.9 with boot 1.5; earlier versions are no longer supported. The current boot 1.5.x version is 1.5.21.

@ClassRule
public static KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, true, TEST_TOPIC);

static {
    embeddedKafka.setKafkaPorts(1234);
}

setKafkaPorts has been available since 1.3.

However, you are properly using the allocated random port in your test

Map<String, Object> senderProps = KafkaTestUtils.senderProps(embeddedKafka.getBrokersAsString());

To get the kafka listener to connect to the embedded broker, you can use.

    System.setProperty("spring.kafka.bootstrap-servers", embeddedKafka.getBrokersAsString());
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • My application is supposed to work with `Kafka 0.9` and I do not see that changing any time soon. Will the versions which you have mentioned work with that? – AlphaEpsilon May 20 '19 at 19:08
  • 0.9 is extremely old; there have been many improvements, especially [KIP-62](https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread) which solves rebalancing issues with slow clients, you should at least try to move to 0.10.2.0. The current version is 2.2.0. 0.9.0.0 is not even mentioned on the [compatibility page](https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix). I have no idea if 1.3.9 will work with 0.9.0.0; you should try it. – Gary Russell May 20 '19 at 19:51
  • Moving to 0.10.2.0 is in talks with the DevOps team at my company and would unfortunately be a slow process. I used my application with 1.3.9 with a local kafka 0.9.0.0 cluster running. The application just prints the Consumer Config values and `o.a.k.c.u.AppInfoParser : Kafka version : 0.11.0.2` and halts. No partition assignment takes place. I guess I will have to work with the older version for now. – AlphaEpsilon May 20 '19 at 20:19
  • You should at least upgrade to 1.0.6.RELEASE which was the last release of 1.0.x for bug fixes. – Gary Russell May 20 '19 at 20:21
  • 1
    Using 1.0.6, I added `System.setProperty(..)` to the `init()` method of the test class but still KafkaListener is connecting to the broker configured in the consumerConfig for the listener container and the test fails. – AlphaEpsilon May 20 '19 at 20:40
0

I think while application context is loaded for test, two beans are creating of type (ProducerFactory and KafkaTemplate), one is with the original configs and second is with test configs, try this use different profile for tests application-test.yml and add bean overriding property 

spring.main.allow-bean-definition-overriding to true.

So that it will override the application beans with test beans and also declare ProducerFactory and KafkaTemplate as beans in test with same name as in application

Ryuzaki L
  • 37,302
  • 12
  • 68
  • 98