1

enter image description here

We have a "Kerberized Kafka cluster" running brokers version Apache Kafka 0.11.This cluster is managed by a different team and we dont have any control over this.

We are now trying to install Kafka Connect cluster on our own K8S cluster.

We were following this compatibility matrix https://docs.confluent.io/current/installation/versions-interoperability.html

According to this, we had to stick with Confluent Platform 3.3.3 images for Schema Registry and Kafka Connect pods since the Brokers version is 0.11

Also we want to enable Connect JMX metrics which was implemented here : https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework

According to the documentation these JMX metrics are supported in Kafka 1.0.0

Questions are:

1) Can we use latest Confluent images (eg: version 5.2.0) with Kafka Brokers version 0.11?

2) With current setup (Confluent 3.3.3) I have enabled all the JMX metrics but we don't see any Kafka Connector based metrics. I see general Kafka Connect metrics, but not metrics per each Connector(eg. Source and Sink metrics).Is this because of the old CP version we use ?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Ashika Umanga Umagiliya
  • 8,988
  • 28
  • 102
  • 185

1 Answers1

1

Kafka Connect Workers that are included in Confluent Platform 3.2 and later are compatible with any Kafka broker that is included in Confluent Platform 3.0 and later

Note: Kafka Connect is not a Confluent Platform feature and is built over plain Apache Kafka API. Therefore, Connect instances using Kafka 1.0 and 2.0 client libraries still work with Kafka 0.11 brokers for basic communication protocols

If you're having Kerberos issues, those are likely unrelated

I see general Kafka Connect metrics, but not metrics per each Connector

In general, there is no such per connector metric such as "number of records read/written to a database", or at least, it would largely depend if the connectors actual implement their own metrics, but yes, the 3.3.x releases are all based on Kafka 0.11, not 1.0, where more Connect API metrics were added

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • I was under the impression that CP 5.x versions use new features of Kafka which available only after version 0.11 – Ashika Umanga Umagiliya Jun 25 '19 at 06:15
  • CP 5.x uses Kafka 2.0, so, yes there are new APIs (like record headers), but the base functionality of Connect should still be able to read and write (key,value) records as before – OneCricketeer Jun 25 '19 at 15:34
  • we tried to install CP 5.2 with Kafka 0.11 brokers and its giving the error ----- "kafka-schema-registry_1 | at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:217) kafka-schema-registry_1 | ... 5 more kafka-schema-registry_1 | Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support DESCRIBE_CONFIGS docker_kafka-schema-registry_1 exited with code 1" ----- seems like this is a missing feature in the brokers. – Ashika Umanga Umagiliya Jul 01 '19 at 07:50
  • That error is from the registry, not Connect, so you might want to post a different question – OneCricketeer Jul 01 '19 at 14:04
  • since Schema Registry + Kafka Connect are components of CP ,I thought the compatibility matrix applies to all the components. – Ashika Umanga Umagiliya Jul 02 '19 at 00:00
  • There is a section of Schema Registry here, yes. https://docs.confluent.io/current/installation/versions-interoperability.html – OneCricketeer Jul 02 '19 at 03:10