1

This is what the developer guide has to say about Dynamic connectors https://docs.confluent.io/current/connect/devguide.html#dynamic-connectors

Not all connectors have a static set of partitions, so Connector implementations are also responsible for monitoring the external system for any changes that might require reconfiguration. For example, in the JDBCSourceConnector example, the Connector might assign a set of tables to each Task. When a new table is created, it must discover this so it can assign the new table to one of the Tasks by updating its configuration. When it notices a change that requires reconfiguration (or a change in the number of Tasks), it notifies the framework and the framework updates any corresponding Tasks

I'm not sure how I'm supposed to do this.

The connector class (extends org.apache.kafka.connect.source.SourceConnector) implements taskConfigs(int), start(Map<String,String>), stop(), config(), version() and taskClass(). It doesn't have a poll() or similar method.

Am I suppose to spawn a thread in start(Map<String,String>) that monitors the external system?

And if changes are detected how do I 'notify the framework'. Is there some API call in the Java SDK that does this or does this means that I have to call stop() followed by start()?

Thanks.

David Breton
  • 169
  • 1
  • 10

1 Answers1

2

Am I suppose to spawn a thread in start(Map<String,String>) that monitors the external system?

Yes, precisely.

how do I 'notify the framework'.

Through the requestTaskReconfiguration() method of the SourceConnector context.

David Breton
  • 169
  • 1
  • 10
  • Then what happens? Does requestTaskReconfiguration() result in a call to stop() on the connector? Does the start method on the connector get called again? Where is this documented? – Ryan Jun 16 '20 at 22:24