1

Installed strimzi

helm repo add strimzi https://strimzi.io/charts/ && helm install strimzi-kafka strimzi/strimzi-kafka-operator

Output:

Name:         strimzi-cluster-operator-587cb79468-hrs9q  
strimzi-cluster-operator with (quay.io/strimzi/operator:0.28.0)

(strimzi-kafka-connect:0.28) I used to build image with following Dockerfile

FROM quay.io/strimzi/kafka:0.28.0-kafka-3.1.0
USER root:root
RUN mkdir -p /opt/kafka/plugins/debezium
COPY ./debezium-connector-mysql/ /opt/kafka/plugins/debezium/
COPY ./debezium-connector-mongodb/ /opt/kafka/plugins/debezium/
COPY ./confluentinc-kafka-connect-elasticsearch/ /opt/kafka/plugins/debezium/
COPY ./mongodb-kafka-connect-mongodb-1.7.0/ /opt/kafka/plugins/debezium/
RUN chown -R kafka:root /opt/kafka
USER 1001

Following is KafkaConnect configuration:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
spec:
  version: 3.1.0
  replicas: 1
  bootstrapServers: my-cluster-kafka-bootstrap:9093
  image: strimzi-kafka-connect:0.28.2
  tls:
    trustedCertificates:
      - secretName: my-cluster-cluster-ca-cert
        certificate: ca.crt
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1

Installed Kafka using https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.28.0

Specifically(kafka-persistent.yaml) with following config:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.1.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.1"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

Following is the status of plugins:

kubectl exec my-connect-cluster-connect-59cfff997b-4kv9b  -it my-connect-cluster-connect -- curl http://localhost:8083/connector-plugins | jq '.' 
[
  {
    "class": "com.mongodb.kafka.connect.MongoSinkConnector",
    "type": "sink",
    "version": "1.7.0"
  },
  {
    "class": "com.mongodb.kafka.connect.MongoSourceConnector",
    "type": "source",
    "version": "1.7.0"
  },
  {
    "class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "type": "sink",
    "version": "11.1.8"
  },
  {
    "class": "io.debezium.connector.mongodb.MongoDbConnector",
    "type": "source",
    "version": "1.8.1.Final"
  },
  {
    "class": "io.debezium.connector.mysql.MySqlConnector",
    "type": "source",
    "version": "1.0.0.Final"
  },
  {
    "class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
    "type": "sink",
    "version": "3.1.0"
  },
  {
    "class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
    "type": "source",
    "version": "3.1.0"
  },
  {
    "class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
    "type": "source",
    "version": "1"
  },
  {
    "class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector",
    "type": "source",
    "version": "1"
  },
  {
    "class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
    "type": "source",
    "version": "1"
  }
]

Everything is working fine when i am creating connector for mysql.

[kafka@my-connect-cluster-connect-59cfff997b-4kv9b kafka]$ curl -i -X POST -H "Accept:application/json" \
>     -H  "Content-Type:application/json" http://localhost:8083/connectors/ \
>     -d  '{
>   "name": "inventory-test-mysql",
>   "config": {
>     "connector.class": "io.debezium.connector.mysql.MySqlConnector",
>     "tasks.max": "1",
>     "database.hostname": "172.17.0.7",
>     "database.port": "3306",
>     "database.user": "root",
>     "database.password": "debezium",
>     "database.server.id": "184054",
>     "database.server.name": "dbserver1",
>     "database.include.list": "inventory",
>     "database.history.kafka.bootstrap.servers": "my-cluster-kafka-bootstrap:9092",
>     "database.history.kafka.topic": "schema-changes-for-inventory",
>     "include.schema.changes": "true"
>   }
> }'
HTTP/1.1 201 Created
Date: Tue, 22 Feb 2022 11:50:14 GMT
Location: http://localhost:8083/connectors/inventory-test-mysql
Content-Type: application/json
Content-Length: 560
Server: Jetty(9.4.43.v20210629)

{"name":"inventory-test-mysql","config":{"connector.class":"io.debezium.connector.mysql.MySqlConnector","tasks.max":"1","database.hostname":"172.17.0.7","database.port":"3306","database.user":"root","database.password":"debezium","database.server.id":"184054","database.server.name":"dbserver1","database.include.list":"inventory","database.history.kafka.bootstrap.servers":"my-cluster-kafka-bootstrap:9092","database.history.kafka.topic":"schema-changes-for-inventory","include.schema.changes":"true","name":"inventory-test-mysql"},"tasks":[],"type":"source"}

Topic and connector created successfully, here is the output:

➜  ~ kubectl exec --tty -i kafka-client-strimzi -- bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --list
__consumer_offsets
__strimzi-topic-operator-kstreams-topic-store-changelog
__strimzi_store_topic
connect-cluster-configs
connect-cluster-offsets
connect-cluster-status
dbserver1
dbserver1.inventory.addresses
dbserver1.inventory.customers
dbserver1.inventory.geom
dbserver1.inventory.orders
dbserver1.inventory.products
dbserver1.inventory.products_on_hand
schema-changes-customers
schema-changes-for-inventory

Connectors status:

[kafka@my-connect-cluster-connect-59cfff997b-4kv9b kafka]$ curl http://localhost:8083/connectors

["inventory-test-mysql"]


[kafka@my-connect-cluster-connect-59cfff997b-4kv9b kafka]$ curl http://localhost:8083/connectors/inventory-test-mysql/status

{"name":"inventory-test-mysql","connector":{"state":"RUNNING","worker_id":"172.17.0.19:8083"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"172.17.0.19:8083"}],"type":"source"}

And i am receiving the schema changes whenever i add or update a record in table of inventory database.

Issue:

This whole setup is not working with either

{
    "class": "com.mongodb.kafka.connect.MongoSinkConnector",
    "type": "sink",
    "version": "1.7.0"
  },
  {
    "class": "com.mongodb.kafka.connect.MongoSourceConnector",
    "type": "source",
    "version": "1.7.0"
  }

or


  {
    "class": "io.debezium.connector.mongodb.MongoDbConnector",
    "type": "source",
    "version": "1.8.1.Final"
  }

Here is the output of curl:

[kafka@my-connect-cluster-connect-59cfff997b-4kv9b kafka]$ curl -i -X POST -H "Accept:application/json" \
>     -H  "Content-Type:application/json" http://localhost:8083/connectors/ \
>     -d  '{
>   "name": "mongodb-connector",
>   "config": {
>       "tasks.max":1,
>       "connector.class":"com.mongodb.kafka.connect.MongoSourceConnector",
>       "connection.uri":"mongodb://root:password@172.17.0.20:27017",
>       "key.converter":"org.apache.kafka.connect.storage.StringConverter",
>       "value.converter":"org.apache.kafka.connect.storage.StringConverter",
>       "key.converter.schemas.enable": "false",
>       "value.converter.schemas.enable": "false",
>       "database":"mydb",
>       "collection":"dataSource"
>   }
> }'
HTTP/1.1 500 Server Error
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: application/json
Content-Length: 137
Connection: close
Server: Jetty(9.4.43.v20210629)

{
"servlet":"org.glassfish.jersey.servlet.ServletContainer-1d98daa0",
"message":"Request failed.",
"url":"/connectors/",
"status":"500"
}

OR with

[kafka@my-connect-cluster-connect-59cfff997b-4kv9b kafka]$ curl -i -X POST -H "Accept:application/json" \
>     -H  "Content-Type:application/json" http://localhost:8083/connectors/ \
>     -d  '{
>   "name": "mongodb-connector",
>   "config": {
>     "connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
>     "mongodb.hosts": "rs0/172.17.0.20:27017,rs0/172.17.0.21:27017",
>     "mongodb.name": "mydb",
>     "mongodb.user": "root",
>     "mongodb.password": "password",
>     "database.whitelist": "mydb[.]*"
>   }
> }'
HTTP/1.1 500 Server Error
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: application/json
Content-Length: 137
Connection: close
Server: Jetty(9.4.43.v20210629)

{
"servlet":"org.glassfish.jersey.servlet.ServletContainer-1d98daa0",
"message":"Request failed.",
"url":"/connectors/",
"status":"500"
}

I would appreciate, if you let me know what am i missing.

If REST is not possible with Strimzi and Mongodb, then what is alternative ?

Is it related to version of Strimzi-Kafka ? Is it related to version of Mongodb plugins ?

Thanks

Faaiz
  • 635
  • 8
  • 18
  • The REST interface you are using is from Apache Kafka, not from Strimzi. So I guess you should check the Kafka Connect logs as to what the issue is and why it does not like your request and with the MongoDb folks to see if your request is correct or not. – Jakub Feb 22 '22 at 13:53
  • 1
    PS: You probably should not copy all the plugins into `/opt/kafka/plugins/debezium/` -> normally each should have its own subdirectory under `/opt/kafka/plugins/`. – Jakub Feb 22 '22 at 13:55
  • @Jakub Thanks for your reply. I built the image again and set the plugins in `/opt/kafka/plugins/` . I get the error in logs ```Caused by: java.lang.NoClassDefFoundError: org/apache/avro/Schema ││ at com.mongodb.kafka.connect.source.MongoSourceConfig.createConfigDef(MongoSourceConfig.java:747)``` To me it seems like version issue – Faaiz Feb 22 '22 at 17:40
  • That suggests that you are probably missing some classes which need to be added. Might be the Avro converters which are not part of Apache Kafka project. – Jakub Feb 22 '22 at 18:18
  • @Jakub, It is resolved. It was version issue that is why he was not able to find certain classes of mongodb plugin. I downgraded the version from `quay.io/strimzi/kafka:0.28.0-kafka-3.1.0` to `FROM strimzi/kafka:0.17.0-kafka-2.4.0` also the version `--kubernetes-version v1.21.10` (see more https://github.com/kubernetes/kubernetes/tags) on minikube. For that i need to recreate cluster on minikube. Need to check till which version up to 0.28.0 is it workable. As i get time, i will post an article with complete working setup soon. Thanks for your feedback and reading the question. – Faaiz Feb 24 '22 at 09:59

0 Answers0