How to run Kafka Connect connectors automatically (e.g. in production)?

Question

Is there a way to automatically load (multiple) Kafka Connect connectors upon the start of Kafka Connect (e.g. in Confluent Platform)?

What I've found out so far:

Confluent Docs state to use the bin/connect-standalone the command for Standalone Mode with a properties file for the worker and for every single connector.

For Distributed Mode you have to run the connector via REST API.

https://docs.confluent.io/current/connect/userguide.html#standalone-mode, https://docs.confluent.io/current/connect/managing/configuring.html#standalone-example

Is there another method, e.g. to include all connectors that should be run in the 'connect-[standalone|distributed].properties' file (similar to providing KSQL queries file in ksql-server.properties) so that they are loaded automatically upon the start of Kafka Connect (e.g. in Confluent Platform)?

Or are the connectors loaded "manually" as described above even in production environments?

Giorgos Myrianthous · Answer 1 · 2019-06-06T10:04:50.773

Normally, you'd have to use the REST API when running Kafka Connect in distributed mode. However, you can use docker compose to script the creation of connectors; @Robin Moffatt has written a nice article about this:

kafka-connect:
  image: confluentinc/cp-kafka-connect:5.1.2
  environment:
    CONNECT_REST_PORT: 18083
    CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect"
    […]
  volumes:
    - $PWD/scripts:/scripts
  command: 
    - bash 
    - -c 
    - |
      /etc/confluent/docker/run & 
      echo "Waiting for Kafka Connect to start listening on kafka-connect ⏳"
      while [ $$(curl -s -o /dev/null -w %{http_code} http://kafka-connect:8083/connectors) -eq 000 ] ; do 
        echo -e $$(date) " Kafka Connect listener HTTP state: " $$(curl -s -o /dev/null -w %{http_code} http://kafka-connect:8083/connectors) " (waiting for 200)"
        sleep 5 
      done
      nc -vz kafka-connect 8083
      echo -e "\n--\n+> Creating Kafka Connect Elasticsearch sink"
      /scripts/create-es-sink.sh 
      sleep infinity

Notes:

In the command section, $ are replaced with $$ to avoid the error Invalid interpolation format for "command" option

sleep infinity is necessary, because we’ve sent the /etc/confluent/docker/run process to a background thread (&) and so the container will exit if the main command finishes.

The actual script to configure the connector is a curl call in a separate file. You could build this into the Docker Compose but it feels a bit yucky.

You could combine both this and the technique above if you wanted to install a custom connector plugin before launching Kafka Connect, e.g. confluent-hub install --no-prompt confluentinc/kafka-connect-gcs:5.0.0 /etc/confluent/docker/run

How to run Kafka Connect connectors automatically (e.g. in production)?

1 Answers1