Spring Boot Kafka Junit with application defined kafka server from docker compose

Question

I have Spring Boot Application with docker-compose,from it I am configuring kafka and other required application.

I have Kafka Producer and Consumer (@kafkaListener) in my application. I have Kafka and JPA Repository configuration from @Bean (not from application.yml file).

I am writing a junit test for kafka producer and consumer, but I want to use application defined kafka (defined in docker-compose).

So in my pom.xml I have exec goal, in which first start the application, then run the test collect report and last stop the application.

Can someone please help me to write Junit for kafka producer and consumer by using application defined kafka. I tried @RunWith(SpringRunner.class), @SpringBootTest, @EmbeddedKafka but always I am getting exception.

  <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <version>1.1.1</version>
            <executions>
                <execution>
                    <id>Create IT environment</id>
                    <phase>pre-integration-test</phase>
                    <goals>
                        <goal>exec</goal>
                    </goals>
                    <configuration>
                        <executable>${basedir}/start-services.sh</executable>
                    </configuration>
                </execution>
                <execution>
                    <id>Generate Cobertura Reports</id>
                    <phase>integration-test</phase>
                    <goals>
                        <goal>exec</goal>
                    </goals>
                    <configuration>
                        <skip>${skip.coverage}</skip>
                        <executable>${basedir}/coverage/report.sh</executable>
                    </configuration>
                </execution>
                <execution>
                    <id>Clean up environment</id>
                    <phase>post-integration-test</phase>
                    <goals>
                        <goal>exec</goal>
                    </goals>
                    <configuration>
                        <executable>${basedir}/stop-services.sh</executable>
                    </configuration>
                </execution>
            </executions>
        </plugin>

Sample Test -

 @RunWith(SpringRunner.class)
 @SpringBootTest
 @EmbeddedKafka(partitions = 1, brokerProperties = {  "listeners=PLAINTEXT://localhost:9092", "port=9092" })
 public class KafkaProcessorITCase {

    private static final String TOPIC = "topic";

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    @Autowired
    private MessageConsumer consumer;

    @Autowired
    private MessageProducer producer;

    @Test
    public void shouldProcessMessage() throws  Exception {

        String message = "Sample message";

        producer.sendMessage(message, TOPIC);
        
        //Assertions
 }
}

Producer Config

  @Bean
  public ProducerFactory<String, String> producerConfig() {
     Map<String, Object> configMap = new HashMap<>();
     configMap.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,      System.getenv("KAFKA_BROKERS"));
     configMap.put(ProducerConfig.ACKS_CONFIG, producerAcks);
     configMap.put(ProducerConfig.RETRIES_CONFIG, retries);
     configMap.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,    StringSerializer.class.getName());
     configMap.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

     return new DefaultKafkaProducerFactory<>(configMap);
  }

ERROR :

    java.lang.IllegalStateException: Failed to load ApplicationContext
at org.springframework.cloud.configuration.CompositeCompatibilityVerifier.verifyDependencies(CompositeCompatibilityVerifier.java:47)
at org.springframework.cloud.configuration.CompatibilityVerifierAutoConfiguration.compositeCompatibilityVerifier(CompatibilityVerifierAutoConfiguration.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:338)
at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:123)
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99)
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:124)
at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:124)
at org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:190)
at org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:132)
at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:248)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:227)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:289)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:291)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:246)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:158)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)

Please share the source code of your test case and the stack trace of the exception. — D-FENS, Dec 04 '21 at 22:47

D-FENS · Answer 1 · 2021-12-04T23:26:32.250

0

Edit: The exception you are getting means that Spring cannot find providers for all the objects annotated with @Autowired. You need to make sure that all autowired objects can be instantiated by a provider method or a bean.

I am by no means a Kafka expert, but looking at this post (sorry, it's only in Russian but take a look at the code anyway), they have a bean for the KafkaTemplate and you used it differently. See their code:

    @Bean
    public KafkaTemplate<String, Greeting> greetingKafkaTemplate() {
        return new KafkaTemplate<>(greetingProducerFactory());
    }

and then they annotated it as @Autowired in KafkaProducer.

Do you have providers for the following variables?

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    @Autowired
    private MessageConsumer consumer;

    @Autowired
    private MessageProducer producer;

By the way, you could look into using Mockito for consumer and producer. I don't know if this works for Kafka though.

edited Dec 04 '21 at 23:26

answered Dec 04 '21 at 23:12

D-FENS

1,438
8
21

Hi roccobaroccoSC, Sorry for the confusion. I have updated the question with error i am getting. when do `mvn claen verify` – ppb Dec 04 '21 at 23:16
It's ok, I will edit my answer. – D-FENS Dec 04 '21 at 23:20
I edited the answer, please take a look. – D-FENS Dec 04 '21 at 23:28
I am doing everything same as mentioned in link but facing same issue. I am not sure why `DefaultKafkaProducerFactory` giving `NullPointer` – ppb Dec 05 '21 at 00:33
I thought we were past that point. Your current exception stack relates to Spring not being able to load ApplicationContext. The NullPointerException is not relevant anymore, is it? – D-FENS Dec 05 '21 at 00:42

Spring Boot Kafka Junit with application defined kafka server from docker compose

1 Answers1