Connecting JDBC sink connector to CrateDB

Question

I am trying to run the jdbc sink connect with cratedb as a sink, as mentioned here. I wanted to run in docker so I created containers for the connector and for my cratedb. But upon running, I keep getting the following error in the logs.

Docker logs of the connector-standalone container:

[2020-08-20 10:34:06,430] ERROR WorkerSinkTask{id=jdbc-sink-connector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179)
org.apache.kafka.connect.errors.ConnectException: Not a valid JDBC URL: "jdbc:postgresql://cratedb:4200/doc?user=crate",
    at io.confluent.connect.jdbc.dialect.DatabaseDialects.extractJdbcUrlInfo(DatabaseDialects.java:175)
    at io.confluent.connect.jdbc.dialect.DatabaseDialects.findBestFor(DatabaseDialects.java:119)
    at io.confluent.connect.jdbc.sink.JdbcSinkTask.initWriter(JdbcSinkTask.java:54)
    at io.confluent.connect.jdbc.sink.JdbcSinkTask.start(JdbcSinkTask.java:46)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:300)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:189)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[2020-08-20 10:34:06,433] ERROR WorkerSinkTask{id=jdbc-sink-connector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:180)

My Docker file:

version: '3.3'

services:

  zookeeper:
    container_name: zookeeper
    image: wurstmeister/zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    ports:
      - '2181:2181'

  kafka:
    container_name: kafka
    image: wurstmeister/kafka:2.12-2.3.0
    env_file:
      - ".env"
    ports:
      - 9092:9092
    depends_on:
      - zookeeper
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  connect-standalone:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: connect-standalone
    ports:
      - 8083:8083
    depends_on:
      - kafka
    volumes:
      - ./connect-input-file:/tmp

  cratedb:
    container_name: cratedb
    image: crate:latest
    ports:
      - "4200:4200"
    volumes:
      - /tmp/crate/01:/data
    command: ["crate",
              "-Cnode.name=cratedb",
              "-Cnode.data=true"]
    environment:
      - CRATE_HEAP_SIZE=2g

My jdbc-sink-connector.properties

name=jdbc-sink-connector
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
topics=metrics
connection.url="jdbc:postgresql://cratedb:4200/doc?user=crate",
auto.create=true

I am quite not sure what I am missing out.

You should be using connect-distributed instead of standalone in an ephemeral environment such as Docker, just like the existing kafka connect containers available — OneCricketeer, Aug 20 '20 at 13:14

score 0 · Answer 1 · answered Sep 01 '20 at 13:27

0

It could be caused by the port 4200 which is the default port for HTTP. The default port for PostgreSQL is 5432

You can also have a look at a docker-compose file we use for testing here

answered Sep 01 '20 at 13:27

mfussenegger

3,931
23
18

Connecting JDBC sink connector to CrateDB

1 Answers1