0

If I use a large task configuration my Kafka Source Connector epics2kafka starts, but no tasks are created. Using a smaller configuration appears to correct the problem. If I move my configuration into a file and pass just the file path along for the task to use, that also works (though no longer scales to distributed nodes as now relying on a local file!). It seems I'm running into a memory limit or internal REST service Post size constraint, or connect topic constraint of some kind. The weird thing is there doesn't appear to be any errors logged. The connector status is "RUNNING", but the task status is an empty array. Using print debug statements I can see the taskConfigs() method always completes, but if using a large config the start() method of the Connect task never is called. The large taskConfigs is 25,500 lines of JSON (about 2.5MB), which is added into the taskConfigs Map.

Similar (but different): Kafka Connector's task list is empty

I'm using version 3.1.0 of Kafka, so I assume the above mentioned issue is different (looks like that issue was due to URL encoding of internal REST Post request).

Ryan
  • 7,499
  • 9
  • 52
  • 61
  • I suggest consuming direct from the Kafka Connect config topic to verify the config actually got stored. You say its 2.5MB, which is too large for a Kafka record, by default. It's not a specific problem for that connector you are using. – OneCricketeer May 20 '22 at 22:17

1 Answers1

0

There is a producer request size limit (for the producer used internally by Kafka Connect to write task configs) and a topic message size limit (for the topic used to store Connect configs) that must be increased. I bumped them up to 5MB by adding max.request.size=5242880 to connect-distributed.properties and max.message.bytes=5242880 to the topic configuration of topic connect-configs.

See: Kafka Users Discussion

Ryan
  • 7,499
  • 9
  • 52
  • 61