0

I'm trying to do cdc with debezium kafka connect with the following setup:

  • Zookeeper
  • Kafka
  • PostgreSQL
  • Kafka Connect Debezium

I'm also configuring and starting them up in that order.

After starting up Kafka Connect Debezium I do a POST to configure it:

{
  "name": "postgres-boxes-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "172.17.0.4",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres",
    "database.dbname" : "postgres",
    "database.server.name": "fullfillment",
    "table.whitelist": "public.boxes"
  }
}

After starting my watcher I see the following warning/error:

WARN [Consumer clientId=consumer-1, groupId=console-consumer-53726] Error while fetching metadata with correlation id 2 : {fullfillment.public.boxes=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Now when I restart Kafka Connect Debezium everything works fine without errors and the cdc works just fine.

I'm now trying to figure out why I have to restart Kafka Connect for it to work and hopefully it can be avoided. If more info is required I'm happy to provide more.

Viktor Baert
  • 686
  • 8
  • 22
  • Are you sure it doesn't work if you don't restart Kafka Connect worker? The `LEADER_NOT_AVAILABLE` is often seen when a new topic gets created - and then subsequently created. – Robin Moffatt Nov 27 '19 at 16:58
  • Quite sure it didn't, However I did try to update records immediately after I've done the setup. Could there be some form of delay? Any case we're trying again tommorow and I'll post an update – Viktor Baert Nov 27 '19 at 17:06
  • So it's just a `WARN` that you saw, which means the task continues executing. If you hit this situation again it would be useful to explain in more detail exactly _what_ isn't working/what the symptoms are. – Robin Moffatt Nov 27 '19 at 17:09
  • 1
    @RobinMoffatt We've retried setting everything up from scratch and didn't run into issues. Thanks for the clarification. – Viktor Baert Nov 29 '19 at 08:00
  • glad it's working. I've posted an answer for you to accept if you agree with it, so that other people finding this post can clearly see what the outcome was. – Robin Moffatt Nov 29 '19 at 09:15

1 Answers1

1

You have to restart the worker in order to pick up the connector plugin when you install it. Other than that, there is no need to restart the Connect worker for the connector to work.

It's just a WARN that you saw, which means the task continues executing. The LEADER_NOT_AVAILABLE is often seen when a new topic gets created - and then subsequently created (If you see LEADER_NOT_AVAILABLE continuously and it doesn't stop then it suggests that the topic has not been automatically created and you may need to create it manually).

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92