Spring boot embedded kafka throws error BeanCreationException

Question

I have a test case with following configured kafka properties:

@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = {Application.class})
@ActiveProfiles("test")
@EnableConfigurationProperties
@EmbeddedKafka(controlledShutdown = true, topics = {"topic1", "topic2", "topic3"})
class Name {}

Here is the build error i get when running the test case above:

Application run failed 
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'embeddedKafka': Invocation of init method failed; nested exception is java.lang.NoSuchFieldError: DEFAULT_SASL_ENABLED_MECHANISMS
Caused by: java.lang.NoSuchFieldError: DEFAULT_SASL_ENABLED_MECHANISMS
    at kafka.server.Defaults$.<clinit>(KafkaConfig.scala:242)
    at kafka.server.KafkaConfig$.<clinit>(KafkaConfig.scala:961)
    at kafka.server.KafkaConfig.LogDirProp(KafkaConfig.scala)
    at org.springframework.kafka.test.EmbeddedKafkaBroker.afterPropertiesSet(EmbeddedKafkaBroker.java:322)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1853)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1790)
    ... 72 common frames omitted

Here is the POM file with spring kafka & kafka-client dependency which is likely causing the error:

         <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
            <version>2.7.7</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.kafka</groupId>
                    <artifactId>kafka-clients</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>3.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka-test</artifactId>
            <version>2.7.7</version>
            <scope>test</scope>
        </dependency>

Solution: Please refer to this link: https://docs.spring.io/spring-kafka/docs/current/reference/html/#update-deps

<dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
            <version>2.7.7</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka-test</artifactId>
            <version>2.7.7</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.8.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.8.1</version>
            <classifier>test</classifier>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.13</artifactId>
            <version>2.8.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.13</artifactId>
            <version>2.8.1</version>
            <classifier>test</classifier>
            <scope>test</scope>
        </dependency>
    ```

score 1 · Accepted Answer · answered Oct 01 '21 at 21:31

1

That spring-Kafka version is not compatible with Apache Kafka client 3.0. You need still coming 2.8 and Spring Boot 2.6.

On the other hand you don’t need newer client even if you use newer broker . Although the story is about embedded testing, so I fully doubt you need to worry about any client, unless it is something transitive from Spring for Apache Kafka…

answered Oct 01 '21 at 21:31

Artem Bilan

113,505
11
91
118

The reason behind excluding and re-importing newer Kafka-clients is because the there is a security vulnerability in my project stemming from the kafka-clients version compiled from spring-kafka. What versions would you recommend for both to resolve this error? – DeeP Oct 03 '21 at 16:17
OK. What is that vulnerability, please? Would you mind to share a link for that and its fix in Apache Kafka? Anyway I don't think it impacts testing somehow. You may consider to have a new `kafka-clients` version for production code, but still use a transitive version for test scope. – Artem Bilan Oct 04 '21 at 13:34
Here's a link to the vulnerability im facing: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-38153. – DeeP Oct 04 '21 at 14:05
Thanks. It said it is fixed in the `2.8.1` as well. Did you try with that one? `3.0.0` client is too aggressive for the current `spring-kafka-2.7.x` – Artem Bilan Oct 04 '21 at 14:08
3

Please see the documentation for how to override the kafka-clients (and associated jars for the embedded broker) version. https://docs.spring.io/spring-kafka/docs/current/reference/html/#update-deps – Gary Russell Oct 04 '21 at 14:11
Yes, I tried 2.8.1 and the issue still persists. – DeeP Oct 04 '21 at 14:12
OK. So, as I said before: try to keep the current `2.7.1` for testing and move production to `2.8.1`. I doubt the security vulnerability affect you somehow when you test against embedded Kafka. – Artem Bilan Oct 04 '21 at 14:14
2

Thanks for the help. Here are the versions that worked for me: Spring-kafka -> 2.7.7 Kafka-clients -> 2.8.1 I had to override the dependencies mentioned in the article @GaryRussell shared. – DeeP Oct 04 '21 at 19:35

Spring boot embedded kafka throws error BeanCreationException

1 Answers1