Two (Kafka) S3 Connectors not working simultaneously

Question

I have a Kafka Connect working in a cluster (3 nodes) with 1 connector (topic -> S3), everything is fine:

root@dev-kafka1 ~]# curl localhost:8083/connectors/s3-postgres/status | jq -r
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   219  100   219    0     0  36384      0 --:--:-- --:--:-- --:--:-- 43800
{
  "name": "s3-postgres",
  "connector": {
    "state": "RUNNING",
    "worker_id": "127.0.0.1:8083"
  },
  "tasks": [
    {
      "state": "RUNNING",
      "id": 0,
      "worker_id": "127.0.0.1:8083"
    },
    {
      "state": "RUNNING",
      "id": 1,
      "worker_id": "127.0.0.1:8083"
    }
  ],
  "type": "sink"
}

But when I created another connector, the task status is always like that:

[root@dev-kafka1 ~]# curl localhost:8083/connectors/s3-postgres6/status | jq -r
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   109  100   109    0     0  14347      0 --:--:-- --:--:-- --:--:-- 15571
{
  "name": "s3-postgres6",
  "connector": {
    "state": "RUNNING",
    "worker_id": "127.0.0.1:8083"
  },
  "tasks": [],
  "type": "sink"
}

I don't know why I did it wrong in configuration that two connectors of the same plugin don't work together, if I stop the connector #1 that is running fine, the connector #2 after restart, work fine. Does anyone know something I should change in configs ?

Try increasing the heap size of Connect? Maybe not enough memory for running more tasks — OneCricketeer, Nov 24 '18 at 23:17
Also, ideally you run Connect on separate machines than the Kafka Brokers — OneCricketeer, Nov 24 '18 at 23:29
It worked! I just created 3 nodes to realocate kafka connect, and increased memory and cpu in these nodes (from aws m4.large to m4.2xlarge) and it's fine now, the connectors tasks are running. Thanks a lot @cricket_007 ! — Paulo Singaretti, Nov 26 '18 at 00:33

score 0 · Accepted Answer · answered Nov 26 '18 at 02:33

Hard to say what exactly the problem could be without searching through the logs, maybe even changing the logging to debug verbosity temporarily, but depending on the connector properties, Kafka Connect can be very memory hungry.

Therefore, I'd suggest running Connect itself on isolated machines from the Kafka brokers, and allowing Connect to take more heap size (the default is 2g in latest versions) by exporting the KAFKA_HEAP_OPTS variable

Two (Kafka) S3 Connectors not working simultaneously

1 Answers1