0

I have a KafkaConnect service running, to which I am able to add a connector via it's REST interface (POST /connectors). The request body looks like this (simplified):

{
  "name": "my-sink",
  "config": {
    "connector.class": "com.streamprocessing.MySinkConnector",
    "topics.regex": "MyComponent\\.MyTopic.*"   //should listen to everything with Prefix "MyComponent.MyTopic"
  }
}

A subsequent GET request to /connectors?expand=info&expand=status then correctly returns me the connector with its task (id 0), both having the state RUNNING.

The task is being executed, but my custom code inside the Task classes' put() is never executed, because the sinkRecords Collection from its signature is always empty and therefore triggers an early return:

public class MySinkTask extends SinkTask {
@Override
public void put(Collection<SinkRecord> sinkRecords) {
   log.info("Start put, batch size: " + sinkRecords.size());
   int batchSize = sinkRecords.size();

   if (batchSize == 0) {
       return;
   }
   // ...
}

This keeps logging me a size of 0 every 10 seconds, I already tested with two different Kafka brokers and verified the reception of my test messages by consuming them separately in both cases.

Possible problem sources which - I am sure - can be excluded:

  • Connection to Kafka broker (topics for config, status and offsets are created successfully -> at least the initial connection is established successfully)
  • Wrong configuration of the connector -> put() function is called successfully and returns because the early return condition is truthy
  • Invalid topic in request body -> I tried with different variations of the regex in topics.regex (including matching everything via ".*") as well as the specific topic (in topics, which should also work according to the documentation

I also tried deploying my source code to our test environment to see if it works there, but that results in the exact same behavior as to when I run everything locally. The sink is supposed to consume messages produced with a specific topic prefix and insert them into a new MongoDB collection.

I do not have any more ideas for what could cause the sinkRecords to be empty, did anybody ever have this issue?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Yasin
  • 72
  • 7
  • Connectors, like any consumer, default to set auto.offset.reset=latest, so are you producing new records **after** starting the connector? Or have you tried overriding that config? – OneCricketeer Mar 09 '23 at 02:16
  • 1
    I did not override that config, but always produced new messages after setting it all up – Yasin Mar 09 '23 at 06:25
  • Not sure, then. You may want to see https://stackoverflow.com/questions/45717658/what-is-a-simple-effective-way-to-debug-custom-kafka-connectors – OneCricketeer Mar 09 '23 at 21:21

0 Answers0