1

Currently we are using a couple of custom connetor plugins for our confluent kafka connect distributed worker cluster. One thing that bothers me for a long time is that kafka connect writes all logs from all deployed connectors to one File/Stream. This makes debugging an absolute nightmare. Is there a way to let kafka connect log the connectors in different Files/Streams?

Via the connect-log4j.properties I am able to let a specific class log to a different File/Stream. But this means that with every additional connector I have to adjust the connect-log4j.properties

Thanks

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Stephan
  • 323
  • 2
  • 11
  • 1
    Have you considered Elasticsearch/Logstash? – Giorgos Myrianthous Sep 06 '18 at 10:00
  • Yes we took this into consideration. And if we dont find any other way this will be our solution. I cannot believe that kafka connect do not have any proper log handling – Stephan Sep 06 '18 at 11:02
  • "proper log handling" meaning what? A separate file for every connector? Have you searched JIRA for this feature or otherwise created it? Seems like a useful idea – OneCricketeer Sep 06 '18 at 12:39

2 Answers2

3

Kafka Connect does not currently support this. I agree that it is not ideal.

One option would be to split out your connectors and have a dedicated worker cluster for each, and thus separate log files.

Kafka Connect is part of Apache Kafka so you could raise a JIRA to discuss this further and maybe contribute it back via a PR?

Edit April 12, 2019: See https://cwiki.apache.org/confluence/display/KAFKA/KIP-449%3A+Add+connector+contexts+to+Connect+worker+logs

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
  • Thanks for the update Robin! I will open a ticket like suggested. And in the meantime we will take the Logstash/Elasticsearch way. – Stephan Sep 07 '18 at 11:39
  • Did you happen to create the JIRA, @Stephan? If so, can you share that, please? Thanks! – avp Apr 11 '19 at 23:30
  • Thanks, @RobinMoffatt :) – avp Apr 12 '19 at 15:56
  • @RobinMoffatt Can we get connector wise log file by adding this `connector context` to `log4j.appender.connectAppender.File=${kafka.logs.dir}/connect.log` ? – Gibbs Jul 13 '21 at 07:39
0

I added this connector.context to log4j.appender.connectAppender.File=${kafka.logs.dir}/connect.log but I was not able to get any connector log separately. Also, the name of the file is not formed with the connector name. It was formed with the variable name - "connector.context"

CSFreak
  • 1