5

I'm trying to run a local kafka-connect cluster using docker-compose. I need to connect on a remote database and i'm also using a remote kafka and schema-registry. I have enabled access to these remotes resources from my machine.

To start the cluster, on my project folder in my Ubuntu WSL2 terminal, i'm running

docker build -t my-connect:1.0.0

docker-compose up

The application runs successfully, but when I try to create a new connector, returns error 500 with timeout.

My Dockerfile

FROM confluentinc/cp-kafka-connect-base:5.5.0

RUN cat /etc/confluent/docker/log4j.properties.template

ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"
ARG JDBC_DRIVER_DIR=/usr/share/java/kafka/

RUN   confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:5.5.0 \
   && confluent-hub install --no-prompt confluentinc/connect-transforms:1.3.2

ADD java/kafka-connect-jdbc /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib/
COPY java/kafka-connect-jdbc/ojdbc8.jar /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib/

ENTRYPOINT ["sh","-c","export CONNECT_REST_ADVERTISED_HOST_NAME=$(hostname -I);/etc/confluent/docker/run"]

My docker-compose.yaml

services:
  connect:
    image: my-connect:1.0.0
    ports:
     - 8083:8083
    environment:
      - CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=http=//schema-registry:8081
      - CONNECT_KEY_CONVERTER=io.confluent.connect.avro.AvroConverter
      - CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=http=//schema-registry:8081
      - CONNECT_BOOTSTRAP_SERVERS=broker1.intranet:9092
      - CONNECT_GROUP_ID=kafka-connect
      - CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
      - CONNECT_VALUE_CONVERTER=io.confluent.connect.avro.AvroConverter
      - CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
      - CONNECT_OFFSET_STORAGE_TOPIC=kafka-connect.offset
      - CONNECT_CONFIG_STORAGE_TOPIC=kafka-connect.config
      - CONNECT_STATUS_STORAGE_TOPIC=kafka-connect.status
      - CONNECT_CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY=All
      - CONNECT_LOG4J_ROOT_LOGLEVEL=INFO
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - CONNECT_REST_ADVERTISED_HOST_NAME=localhost

My cluster it's up

~$ curl -X GET http://localhost:8083/
{"version":"5.5.0-ccs","commit":"606822a624024828","kafka_cluster_id":"OcXKHO7eT4m9NBHln6ACKg"}

Connector call

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d
{
    "name": "my-connector",
    "config":  
    { 
    "connector.class" : "io.debezium.connector.oracle.OracleConnector",
    "tasks.max": "1",
    "database.user": "user", 
    "database.password": "pass",    
    "database.dbname":"SID",
    "database.schema":"schema",
    "database.server.name": "dbname",   
    "schema.include.list": "schema",    
    "database.connection.adapter":"logminer",   
    "database.hostname":"databasehost",
    "database.port":"1521"
   }
}

Error

{"error_code": 500,"message": "IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout"}

## LOG
connect_1  | [2021-07-01 19:08:50,481] INFO Database Version: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
connect_1  | Version 19.4.0.0.0 (io.debezium.connector.oracle.OracleConnection)
connect_1  | [2021-07-01 19:08:50,628] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
connect_1  | [2021-07-01 19:08:50,643] INFO AbstractConfig values:
connect_1  |  (org.apache.kafka.common.config.AbstractConfig)
connect_1  | [2021-07-01 19:09:05,722] ERROR IO error forwarding REST request:  (org.apache.kafka.connect.runtime.rest.RestClient)
connect_1  | java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Connect Timeout

Testing the connection to the database

$ telnet databasehostname 1521 Trying <ip>... Connected to databasehostname

Testing connection to kafka broker

$ telnet broker1.intranet 9092 Trying <ip>... Connected to broker1.intranet

Testing connection to remote schema-registry

$ telnet schema-registry.intranet 8081 Trying <ip>... Connected to schema-registry.intranet

What am I doing wrong? Do I need to configure something else to allow connection to this remote database?

Malkath
  • 77
  • 3
  • 13

1 Answers1

2

You need to set correctly rest.advertised.host.name (or CONNECT_REST_ADVERTISED_HOST_NAME, if you’re using Docker). This is how a Connect worker communicates with other workers in the cluster.

For more details see Common mistakes made when configuring multiple Kafka Connect workers by Robin Moffatt.

In your case try to remove CONNECT_REST_ADVERTISED_HOST_NAME=localhost from compose file.

Iskuskov Alexander
  • 4,077
  • 3
  • 23
  • 38
  • I've already tried this, but when I try to upload the compose with "network_mode" = host I get the error "connect_1 | sh: 1: export: 172.17.0.1: bad variable name" – Malkath Jul 07 '21 at 12:34
  • 1
    I figure it out. It was silly. - CONNECT_GROUP_ID was with the same name as another instance that ran in the same environment and used the same configuration topics. Changing this, it worked! Thanks anyway. – Malkath Jul 07 '21 at 13:31
  • @Malkath, you're welcome! If my answer was helpful. please vote up – Iskuskov Alexander Jul 08 '21 at 09:07