Mount (Volumes) not working when embedding Kafka connect into Ksqldb-server

Question

The "volumes" mapping does not seem to work when trying to embed Kafka connector in Ksqldb Server

Below is my docker file

ksqldb-server:
        image: confluentinc/ksqldb-server:0.18.0
        hostname: ksqldb-server
        container_name: ksqldb-server
        depends_on:
          - broker
          - schema-registry
        ports:
          - "8088:8088"
        volumes:
          - "/usr/local/share/kafka/connectors/debezium-debezium-connector-mysql/lib:/usr/share/kafka/plugins/"
        environment:
          KSQL_LISTENERS: "http://0.0.0.0:8088"
          KSQL_BOOTSTRAP_SERVERS: "broker:9092"
          KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
          KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
          KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
          # Configuration to embed Kafka Connect support.
          KSQL_CONNECT_GROUP_ID: "ksql-connect-cluster"
          KSQL_CONNECT_BOOTSTRAP_SERVERS: "broker:9092"
          KSQL_CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.storage.StringConverter"
          KSQL_CONNECT_VALUE_CONVERTER: "io.confluent.connect.avro.AvroConverter"
          KSQL_CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
          KSQL_CONNECT_CONFIG_STORAGE_TOPIC: "_ksql-connect-configs"
          KSQL_CONNECT_OFFSET_STORAGE_TOPIC: "_ksql-connect-offsets"
          KSQL_CONNECT_STATUS_STORAGE_TOPIC: "_ksql-connect-statuses"
          KSQL_CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
          KSQL_CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
          KSQL_CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
          KSQL_CONNECT_PLUGIN_PATH: "/usr/share/kafka/plugins"

I see the below error when I am trying to create a source using the connector

{ "error_code" : 500, "message" : "Failed to find any class that implements Connector and which name matches io.debezium.connector.mysql.MySqlConnector, available connectors are: ..........}" }

I do have the required jar files installed locally in the folder "/usr/local/share/kafka/connectors/debezium-debezium-connector-mysql/lib" as mapped in the volumes in the docker file above.

What is it that I am missing?

Much appreciated

Also, the mounts work fine. The plugin scanner/path config might be incorrect, and you should see logs that say what plugins are scanned as the container starts — OneCricketeer, Jul 20 '21 at 18:02
"ls -a" results in the below - . .. antlr4-runtime-4.7.2.jar debezium-connector-mysql-1.1.0.Final.jar debezium-ddl-parser-1.1.0.Final.jar mysql-connector-java-8.0.16.jar debezium-api-1.1.0.Final.jar debezium-core-1.1.0.Final.jar mysql-binlog-connector-java-0.19.1.jar — Gautam T Goudar, Jul 20 '21 at 18:49
It works with non-embedded Connect server. I can log into the docker container and navigate the folders. (docker exec -it ksqldb-server bash). I am unable to find the file that has the link to the plugin folders. connect-distributed.properties seems missing in the ksqldb-server container. — Gautam T Goudar, Jul 20 '21 at 18:53
Ksqldb does not use connect-distributed as its propeties file — Ran Lupovich, Jul 20 '21 at 19:00
The connect.properties files is in /etc/ksqldb folder inside the docker container. The path to plugins is as follows - what I have in the docker file plugin.path=/usr/share/kafka/plugins. So this part seems to be right in the docker file. — Gautam T Goudar, Jul 20 '21 at 19:08

Ran Lupovich · Answer 1 · 2021-07-20T19:07:03.340

0

After downloading a connector zip file extract their files into the plugin path in your worker configuration (e.g. connect-distributed.properties) using the plugin.path configuration property. As an example, let’s assume you have downloaded the Debezium MySQL connector archive and extracted its contents to /kafka/connect/debezium-connector-mysql. Then you’d specify the following in the worker config:

plugin.path=/kafka/connect

So in your case you mounting to an inner path, the connector plugin needs to be inside another folder and not directly the plugin path mount

edited Jul 20 '21 at 19:07

answered Jul 20 '21 at 18:58

Ran Lupovich

1,655
1
6
13

I tried the below - still seem to get the same error. volumes: - "/usr/local/share/kafka/connectors/:/usr/share/kafka/plugins/" – Gautam T Goudar Jul 20 '21 at 19:10
It seems like your jars are inside lib, and directory not called debezium-connector-mysql-1.6.0.Final-plugin, try to download the connector again , remove other files from the directory and just extract it there – Ran Lupovich Jul 20 '21 at 19:34
Keep the directory tree as it is in tar file , do not change anything – Ran Lupovich Jul 20 '21 at 19:35
I re-installed the plugin and changed the docker compose file to back it was. I still see the same error. volumes: - "/usr/local/share/kafka/connectors/debezium-debezium-connector-mysql/lib/:/usr/share/kafka/plugins/" Also realised - the connect.properties is generated based on the environment variables specified in the docker compose file. So this line => KSQL_CONNECT_PLUGIN_PATH: "/usr/share/kafka/plugins" is where we are asking KsqlDB server to look for connector and in volumes mapping it to files on the local machine running docker. – Gautam T Goudar Jul 20 '21 at 20:13
It is not what I advised you to do...there are clear instructions in debezium site on how to install the connectors, try following along – Ran Lupovich Jul 20 '21 at 20:42
I am trying to "embed" Kafka Connect support into KSQLDB server, so that I can create materialized cache. So even if I take off the debezium connector, I run into the same problem if I use other connectors. So my problem is specific to the docker mount/mapping and not related to debezium. – Gautam T Goudar Jul 21 '21 at 15:03

score 0 · Answer 2 · answered Aug 02 '21 at 18:42

The below mapping worked for me

volumes:
          - "./confluent-hub-components/:/usr/share/kafka/plugins/"

The possible answer is - for the folder "/usr/local/share/kafka/connectors/" perhaps, there is a permission related issue and hence the mapping was not working in the docker compose file. With the new folder as above (which is not in /usr root folder), mapping works.

Mount (Volumes) not working when embedding Kafka connect into Ksqldb-server

2 Answers2