1

How to start multiple Kafka connectors in a Kafka Connect world within a single distributed worker(running on 3 different servers)?

Right now I have a need of 4 Kafka Connectors in this distributed worker(same group.id).

Currently, I am adding one connector at a time using following curl command.

curl -X POST -H "Content-type: application/json" -d '<my_single_connector_config>' 'http://localhost:8083/connectors'

Issue:

  • For each new connector I add, previous/existing connector(s) restarts along with new connector.

Question:

  • How should I start/create all these new connectors with one REST call in a distributed worker mode?
  • Is there any way to have all connector configs in a single REST call, like an array of connector configs?

I tried to search for the same but didn't come across any workaround for this.

Thanks.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
suraj_fale
  • 978
  • 2
  • 21
  • 53

1 Answers1

4

For each new connector I add, previous/existing connector(s) restarts along with new connector.

Yes, that's the current behaviour of Kafka Connect. For further discussion see:

How should I start/create all these new connectors with one REST call in a distributed worker mode? Is there any way to have all connector configs in a single REST call, like an array of connector configs?

You can't do it in a single REST call

If you want to isolate your connectors from each other when creating/updating them, you can just run multiple distributed clusters.

So instead of 1 distributed Connect cluster running 3 connectors, you could have 3 distributed Connect clusters each running 1 connector.

Remember in practice a 'distributed Cluster' could just be of a single node, and indeed could all run on the same machine. You'd scale out for resilience and throughput capacity.

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
  • Thanks, that's what I thought. I hope in future they will have some solution or workaround for this issue. – suraj_fale Nov 28 '18 at 20:26
  • 2
    If you open a JIRA on Apache Kafka (https://issues.apache.org/jira/projects/KAFKA/issues), there's a chance we'll improve the REST API and support requests to create multiple connectors at once. Sounds like a good idea and even easy to do. In addition, we are working avoiding all the restarts when you create a new connector. – Gwen Shapira Nov 28 '18 at 23:36
  • Thanks, I created JIRA as Improvement here https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-7835 – suraj_fale Jan 25 '19 at 16:55