Error creating bean with name 'embeddedKafka' versions of Kafka

Question

getting:

Error creating bean with name 'embeddedKafka'

Caused by: java.lang.ClassNotFoundException: com.codahale.metrics.Reservoir

 <kafka.version>2.8.1</kafka.version>
 <spring.kafka.version>2.8.1</spring.kafka.version>

springboot version: 2.6.3 - can't be changed

        <dependency>
            <groupId>org.springframework.integration</groupId>
            <artifactId>spring-integration-test</artifactId>

            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.springframework.integration</groupId>
            <artifactId>spring-integration-kafka</artifactId>

        </dependency>
      
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
             <version>${spring.kafka.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <version>${spring.boot.version}</version>
            <scope>test</scope>
          
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka-test</artifactId>
            <version>${spring.kafka.version}</version>
            <scope>test</scope>
        </dependency>


        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <classifier>test</classifier>
            <scope>test</scope>
            <version>${kafka.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.13</artifactId>
            <classifier>test</classifier>
            <scope>test</scope>
            <version>${kafka.version}</version>
        </dependency>

Where is the conflict? Maybe is some other dependency?

Artem Bilan · Accepted Answer · 2022-07-25T20:28:11.330

0

Please, follow instructions in the docs: https://docs.spring.io/spring-kafka/docs/2.8.1/reference/html/#update-deps.

Consider to use the latest Spring for Apache Kafka in that generation: https://spring.io/projects/spring-kafka#learn.

It looks like something brings incompatible Zookeeper version and we have a special comment for that in the docs for the mentioned version:

<!-- needed if downgrading to Apache Kafka 2.8.1 -->
<exclusions>
    <exclusion>
        <groupId>org.apache.zookeeper</groupId>
        <artifactId>zookeeper</artifactId>
    </exclusion>
</exclusions>

UPDATE

This pom works for me:

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.6.3</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.example</groupId>
<artifactId>so-73113638</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>so-73113638</name>
<description>so-73113638</description>
<properties>
    <java.version>1.8</java.version>
    <kafka.version>2.8.1</kafka.version>
</properties>
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

I just provide a kafka.version property and rely on the Spring Boot feature to manage deps for me.

edited Jul 25 '22 at 20:28

answered Jul 25 '22 at 19:11

Artem Bilan

113,505
11
91
118

That's for 2.9.x + when downgrading from the 3.0.x clients and later; we only pull it in as an `api()` in the gradle build. – Gary Russell Jul 25 '22 at 19:16
Yeah... We don't say that in the doc, only this `When using 2.8.1, you also need to exclude zookeeper dependency from spring-kafka-test.`. Anyway that's strange to use `spring-kafka-2.8.1`, when there is already `2.8.8`. Also not clear the reason behind using Apache Kafka `2.8.1`, when it is much better to rely on whatever Spring Boot brings for us... – Artem Bilan Jul 25 '22 at 19:43
zookeeper 3.6.3 comes from org.springframework.kafka:spring-kafka-test:jar:2.8.1:test when I ran dependency tree – YerivanLazerev Jul 25 '22 at 19:45
Why don't rely on whatever that Spring Boot version brings for you: https://github.com/spring-projects/spring-boot/blob/v2.6.3/spring-boot-project/spring-boot-dependencies/build.gradle#L1672 ? – Artem Bilan Jul 25 '22 at 19:46
I just removed the spring.kafka.version property and fail on same, tree shows org.springframework.kafka:spring-kafka:jar:2.8.2 – YerivanLazerev Jul 25 '22 at 19:50
Correct. And now remove `kafka.version` as well. Plus respective manual deps. Just rely on the transitive deps from `spring-kafka-test` – Artem Bilan Jul 25 '22 at 19:51
kafka.version version is required there for org.apache.kafka:kafka-clients:jar:test and org.apache.kafka:kafka_2.13:jar:test is missing – YerivanLazerev Jul 25 '22 at 19:55
Where? If you remove `kafka-clients` and `kafka_2.13`, the proper version should be brought by the `spring-kafka-test` transitively. – Artem Bilan Jul 25 '22 at 19:59
See its pom: https://repo1.maven.org/maven2/org/springframework/kafka/spring-kafka-test/2.8.2/spring-kafka-test-2.8.2.pom. Or you just want to downgrade to Kafka Client from that default `3.0.0` just because it has some problem on Windows? Then you need to follow the docs I've mentioned in my answer: https://docs.spring.io/spring-kafka/docs/2.8.1/reference/html/#update-deps. It just does exactly what you'd like to hear from us, with only difference that it does not override Spring for Apache Kafka version. – Artem Bilan Jul 25 '22 at 20:03
I removed those 2 dependencies and now fail on ClassNotFoundException: io.micrometer.core.instrument.binder.system.DiskSpaceMetrics – YerivanLazerev Jul 25 '22 at 20:06
See an UPDATE in my answer. So, I rely on `spring-kafka` version from Spring Boot and just override a version property for Kafka Client. The rest of "magic" is done for me by Spring Boot dependency management. – Artem Bilan Jul 25 '22 at 20:29

score 0 · Answer 2 · answered Jul 25 '22 at 19:13

0

You need to use Zookeeper 3.6.3; Kafka no longer pulls it in transitively.

answered Jul 25 '22 at 19:13

Gary Russell

166,535
14
146
179

to which which minimal versions of kafka.version and spring.kafka.version with springboot 2.6.3 I can take in order not to use zookeeper? – YerivanLazerev Jul 25 '22 at 19:42
You cannot get rid of it. You use embedded Kafka which is essentially Kafka Broker by itself. And that one requires Zookeeper to run. – Artem Bilan Jul 25 '22 at 19:45
Newer brokers (3.x, I think) can be run in Raft mode (no ZK needed), but it might still be experimental, I am not sure; we haven't switched the embedded broker to RAFT mode yet. – Gary Russell Jul 25 '22 at 20:57
Thanks , Gary, for education! Anyway see my answer: looks like that works for downgrading – Artem Bilan Jul 26 '22 at 11:45

Error creating bean with name 'embeddedKafka' versions of Kafka

2 Answers2