4

Within the remote kafka cloud cluster, the kafka brokers will be updated to a new version (5.1) and therefore a new kafka protocol applies.

Now I should update my kafka client to being able to connect. Right now I am using following kafka relevant dependencies in my spring-boot application:

<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-parent</artifactId>
<version>1.3.8.RELEASE</version>

<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-kafka</artifactId>
<version>2.0.1.RELEASE</version>

<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<version>1.0.3.RELEASE</version>

<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>1.0.3.RELEASE</version>

<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.9.0.1</version>

<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.9.0.1</version>

I already searched for an appropriate compatibility matrix for kafka-clients and kafka-brokers.

  • Does anyone have an idea how to upgrade successfully ?
  • Does anyone have a comptability matrix for this dependencies?
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Yasé Dogan
  • 53
  • 1
  • 1
  • 4

3 Answers3

10

From https://github.com/spring-cloud/spring-cloud-stream/wiki/Kafka-Client-Compatibility

Compatibility matrix

+--------+--------------+------------------+---------------+-------------+
| Spring |  Spring for  |      Spring      | kafka-clients |    Kafka    |
|  Cloud | Apache Kafka |    Integration   |               |    Broker   |
| Stream |              | for Apache Kafka |               |             |
+--------+--------------+------------------+---------------+-------------+
| 2.1.x  | 2.2.x        | 3.1.x            | 2.0.0,        | 2.0.0,      |
|        |              |                  | 1.1.x,        | 1.1.x,      |
|        |              |                  | 1.0.x(*)      | 1.0.x,      |
|        |              |                  |               | 0.11.0.x(*) |
+--------+--------------+------------------+---------------+-------------+
| 2.0.x  | 2.1.x        | 3.0.x            | 1.1.x,        | 1.1.x,      |
|        |              |                  | 1.0.x(*)      | 1.0.x,      |
|        |              |                  |               | 0.11.0.x(*) |
+--------+--------------+------------------+---------------+-------------+
| 1.3.x  | 1.3.x,       | 2.3.x,           | 0.11.0.x(**), | 0.10.x.x    |
|        | 1.2.x,       | 2.2.x,           | 0.10.2.x      | or higher   |
|        | 1.1.x        | 2.1.x            |               |             |
+--------+--------------+------------------+---------------+-------------+
| 1.2.x  | 1.2.x,       | 2.2.x,           | 0.10.1.x      | 0.10.x.x    |
|        | 1.1.x        | 2.1.x            |               | or higher   |
+--------+--------------+------------------+---------------+-------------+

Other sources

Anton Yuriev
  • 580
  • 8
  • 10
5

According to the documentation:

Previously, the general rule was that the Kafka broker a client interacted with was required to be a newer version than the client. Now, newer versions of the Java clients and other clients that support KIP-35 such as librdkafka are able to fall back to older request types or throw appropriate errors if functionality is not available.

There's a link to the confluent matrix on the Spring for Apache Kafka project page (along with spring-kafka/kafka-clients compatibility).

Sergey
  • 3,253
  • 2
  • 33
  • 55
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Can you update your link with latest version? The confluent link has not been updated since 2019 and on maven I see that kafka-clients library already has 3.4.0 version released as of today. – theprogrammer Apr 16 '23 at 02:40
  • 1
    I am not aware of any updated matrix, reach out to the Kafka community, e.g. on their mailing list. Let me know if you are successful. – Gary Russell Apr 16 '23 at 14:00
2

Bidirectional Client Compatibility is now supported, you don't need to worry about the compatibility matrix anymore, for KIP-35 enabled clients, any version are good, KIP-35 is released from Broker protocol - 0.10.0, Java clients - 0.10.2

refer:

https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix

https://www.confluent.io/blog/upgrading-apache-kafka-clients-just-got-easier/

LIU YUE
  • 1,593
  • 11
  • 19