2

Trying to figure out something about consumer groups in Kafka Connect.

When I create a connector, behind the scenes a consumer group is actually created right?
But when you go to the actual API and do a get request on the connector you will be able to see the connector is actually a thread with a status (so it can fail).

My question is - what is this thread's purpose? I was sure a consumer group (and a connector) is just a logical component...

enter image description here

omer bar lev
  • 341
  • 1
  • 4
  • 11

1 Answers1

3

Creating a connector does not necessarily create a consumer group. COnsumer groups are not really related to the rest of your question so I'll focus answering "What is the Kafka Connect connector thread purpose?"

When Kafka Connect starts a connector, it first launches a "connector" thread. This is the entry you highlighted in the image.

The connector thread has multiple duties:

  • Manage task configurations. This include validation, generating configuration for each task and reconfiguration when requested.

  • Handle global state. The Connect runtime ensures a single instance of the Connector thread runs so it can be used to perform actions that should be done once. For example, in MirrorMaker2 the Connector thread regularly list topics in the source cluster. If it detects new topics, it reconfigures tasks to mirror it.

See the associated methods in the Connector API.

Once the Connector thread is running, the Connect runtime can use it to generate task configurations and start tasks. Task threads do the work of copying data between the external system and Kafka.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • Hi, thanks. When is a connector not a consumer group? I was sure the connector is just an abstraction for the consumer group. – omer bar lev Jul 26 '21 at 11:46
  • Why would a connector be a consumer group? Where did you see this? Are you sure you're talking about Connectors? For example, if you have a source connector importing data from a database into Kafka, why should this use a consumer group, it could only be running producers! – Mickael Maison Jul 26 '21 at 12:01
  • 1
    oh right, I was talking about Sink Connectors, aren't they always form a consumer group? Selected answer over here (bullet 2) - https://stackoverflow.com/questions/41720037/kafka-connect-creating-a-new-connector-in-distributed-mode-is-creating-new-grou – omer bar lev Jul 26 '21 at 12:08
  • @omerbarlev Sink connectors do, yes, but that's not really relevant to the question you asked initially – OneCricketeer Jul 26 '21 at 14:56
  • Indeed, it was just another question I had following the answer since I was surprised. – omer bar lev Jul 26 '21 at 15:24