Kafka connect JDBC source connector not working

Question

Hello Everyone, I am using Kafka JDBC Source connector using for postgres. Following is my connector configuration. Some how it is not bringing any data. What is wrong in this configuration?

{
    "name": "test-connection",
    "config": {
       "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
       "mode": "timestamp",
       "timestamp.column.name": "TEST_DT",
       "topic.prefix": "test",
       "connection.password": "xxxxxx",
       "validate.non.null": "false",
       "connection.user": "xxxxxx",
       "table.whitelist": "test.test",
       "connection.url": "jdbc:postgresql://xxxx:5432/xxxx?ssl=true&stringtype=unspecified",
       "name": "test-connection"
},
  "tasks": [],
  "type": "source"
}

Do I need to create the topic or does it get generated automatically?

I expect the data to be flowing based on the example but the data is not flowing.Following is the log I see in the kafka connect. But, no data is flowing in.

Log

[2019-07-07 20:52:37,465] INFO WorkerSourceTask{id=test-connection-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-07-07 20:52:37,465] INFO WorkerSourceTask{id=test-connection-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)

Answer 1

I faced exactly the same problem and there was no error in the log, though I was adding/modifying the records in postgres and it was not sending any messages. Was getting the same log messages in INFO mode as you mentioned. Here I resolved it and possibly one or all of these might be causing this. So please check what was the issue at your end. If it solves your issue, please accept this as the answer.

1. "table.whitelist" : "public.abcd" <-- This property you ensure you give the schema name also explicitly e.g. I gave "public" as my "abcd" table was in this schema.

2. Usually the Database when we run (e.g. via Docker) the timezone is in UTC mode and if you are in a timezone more than that then while querying the data internally it would put the filter condition such a way that your data is filtered out. To overcome the best way is your timestamp column should be "timestamp with timezone" this solved my issue. Another variation I did was i inserted the data and gave the value of this column as "now() -interval '3 days'" to ensure the data is old and immediately it flow to Topic. Well, best is to give timestamp with timezone instead of this hack.

3. Finally another possible solution could be while giving the connector config you tell what timezone is your postgres db is. You may google for the property. As point-2 solved my issue so i didn't try this.

   CREATE TABLE public.abcd ( id SERIAL PRIMARY KEY, title VARCHAR(100) not NULL, update_ts TIMESTAMP with time zone default now() not null );

My config which worked. in case needed.

{
  "name": "jdbc_source_connector_postgresql_004",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url": "jdbc:postgresql://192.168.99.116:30000/mydb",
    "connection.user": "sachin",
    "connection.password": "123456",
    "topic.prefix": "thesach004_",
    "poll.interval.ms" : 1000,
    "mode":"timestamp",
    "table.whitelist" : "public.abcd",
    "timestamp.column.name": "update_ts",
    "validate.non.null": false,
    "transforms":"createKey,extractInt",
    "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey",
    "transforms.createKey.fields":"id",
    "transforms.extractInt.type":"org.apache.kafka.connect.transforms.ExtractField$Key",
    "transforms.extractInt.field":"id"
  }
}

-$achin

Answer 2

Do I need to create the topic or does it get generated automatically?

it generates automatically with the "test" prefix you set in "topic.prefix": "test"

so your topic is called "testtest-connection" or "testtest.test"

it is possible that you are using Avro schema, if so, you have to consume the topic with Avro consumer.