How Do I Override the Bootstrap Servers' Configuration Dynamically when Using Kafka Testcontainers in Spring Boot?

Question

I'm working on integration tests using Kafka Testcontainers. The container starts up, and reads the bootstrap servers' configuration that is set in the application.yml.

Since the Testcontainers port is created dynamically, I'm using kafkaContainer.getBootstrapServers() to get the socket created by the test container.

According to the Spring Kafka documentation, and several online articles, the DynamicPropertyRegistry class can be used to override the existing configuration.

Here is the code I am using to override the boostrap servers configuration.

@DynamicPropertySource
static void kafkaProperties(DynamicPropertyRegistry registry) {
    registry.add("spring.kafka.bootstrap-servers", kafkaContainer::getBootstrapServers);
}

However, the code above seems to have no effect, since the output of the test displays several lines of

Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.

and then eventually fails with this error message:

org.springframework.kafka.KafkaException: Send failed

at org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:731)
at org.springframework.kafka.core.KafkaTemplate.observeSend(KafkaTemplate.java:691)
at org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:514)
at com.zeal.accelerator.kafka.service.KafkaService.sendMessage(KafkaService.java:38)
at com.zeal.accelerator.kafka.producer.Producer.sendAvroMessage(Producer.java:24)
at com.zeal.integration.accelerator.kafka.ProducerIT.verify_that_expected_event_is_successfully_sent(ProducerIT.java:96)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
Caused by: org.apache.kafka.common.errors.TimeoutException: Topic test-topic not present in metadata after 60000 ms.

This is the test class.

@Slf4j
@Testcontainers
@ActiveProfiles({"test"})
@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = KafkaApplication.class)
class ProducerIT {

private final static String TEST_TOPIC = "test-topic";

private TestAvroMessage expectedPayload;

private final CountDownLatch latch = new CountDownLatch(1);

@Autowired
private KafkaService kafkaService;

@Autowired
private Producer kafkaProducer;

@Autowired
private KafkaProperties kafkaProperties;

@Container
private static final KafkaContainer kafkaContainer =
        new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:latest"))
                .withEmbeddedZookeeper();

@DynamicPropertySource
static void kafkaProperties(DynamicPropertyRegistry registry) {
    registry.add("spring.kafka.bootstrap-servers", kafkaContainer::getBootstrapServers);
}

@Test
void verify_that_expected_event_is_successfully_sent() throws ExecutionException, InterruptedException {

    ReflectionTestUtils.setField(kafkaService, "producerTopic", TEST_TOPIC);
    kafkaProperties.setBootstrapServers(List.of(kafkaContainer.getBootstrapServers()));



    TestAvroMessage actual = TestAvroMessage.newBuilder()
            .setId("aDemoId")
            .setName("test")
            .setProcessed(true)
            .setCost(125000.00)
            .setLevel(5)
            .build();

    kafkaProducer.sendAvroMessage(actual);

    latch.await(10, TimeUnit.SECONDS);

    assertThat(actual).isEqualTo(expectedPayload);
}

@KafkaListener(topics = TEST_TOPIC,
        groupId = "${test.kafka.group-id}",
        containerFactory = "eventListenerContainerFactory")
void messageListener(TestAvroMessage event,
                     @Header(value = KafkaHeaders.RECEIVED_KEY, required = false) String messageKey,
                     @Header(KafkaHeaders.RECEIVED_PARTITION) String partitionId,
                     @Header(KafkaHeaders.OFFSET) long offset,
                     @Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
                     @Headers Map<String, Object> headers) throws InterruptedException {

    latch.countDown();

    expectedPayload = event;
}
}

I'm looking for help solving this. Architectural requirements include use of Testcontainers and precludes the use of EmbeddedKafka. Thanks!

score 1 · Answer 1 · answered Feb 01 '23 at 18:39

We (Kent Hervey and I) found the answer as below:

The @DynamicPropertySource was definitely being executed.
The property spring.kafka.bootstrap-servers was incorrect. We were not able to find this in the Spring Kafka documentation. We used the debugger to step through and figure out which property key was available.
After changing the property key, the test was able successfully connect to the Kafka broker running in the Testcontainer.
What we’re working on now is a schema registry Testcontainer (it doesn’t exist, so we are extending the GenericContainer).
We are still calling the same method on the KafkaTemplate, since that is actually called in the service class. spring.kafka.properties.bootstrap.servers

The following code reflects that changes we made to set the Kafka Bootstrap servers dynamically.

@DynamicPropertySource
static void kafkaProperties(DynamicPropertyRegistry registry) {
    registry.add("spring.kafka.properties.bootstrap.servers", kafkaContainer::getBootstrapServers);
}

How Do I Override the Bootstrap Servers' Configuration Dynamically when Using Kafka Testcontainers in Spring Boot?

1 Answers1