I'm creating a Kafka Connector that loads a list of elements to be requested. This list of elements is being distributed between the different tasks.
Lets say we have 100 elements, and 4 tasks.max configured, each task will have 25 elements to work.
My concern comes from the fact that tasks.max indicates a MAXIMUM of tasks, not a stable number. Kafka can stop some tasks if he needs it.
I would like to recalculate the list of elements inside each task after each full iteration, just to be sure that we are not loosing any info if a task is stopped. But to do this, I need to know from inside the SourceTask implementation how many active tasks are running, and which task number I am.
Currently I am giving the task number and tasks.max value to each task from the SourceConnector implementation, inside the taskConfigs method I initialize each tasks.
Maybe Kafka re throws this method each time it's escalating the task instances (so my problem would not exist)... but I can't fin any info about this.
Thanks in advance for your help!
