5

I am using confluent kafka connect service but it is not writing logs in /var/log/kafka . How to configure it so that it writes the logs in /var/log/kafka ?

Currently /var/log/kafka only has following log files -

-rw-r--r-- 1 cp-kafka confluent     0 Sep 20 14:51 kafka-request.log
-rw-r--r-- 1 cp-kafka confluent     0 Sep 20 14:51 kafka-authorizer.log
-rw-r--r-- 1 cp-kafka confluent  1622 Nov 13 15:43 log-cleaner.log
-rw-r--r-- 1 cp-kafka confluent  7611 Nov 13 20:57 state-change.log
-rw-r--r-- 1 cp-kafka confluent  1227 Nov 14 11:13 server.log
-rw-r--r-- 1 cp-kafka confluent 16683 Nov 14 11:13 controller.log

When checked further , I found logs written to /var/log/messages (which I do not want) . Have a look at below connect-log4j.properties :

log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERROR

I am struggling to find these basic stuff in Confluent docs . Please help !

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
Tony
  • 671
  • 1
  • 9
  • 29

1 Answers1

5

Currently, all logging seems to be redirected to standard output. To redirect logs to a file you need to change the content of /etc/kafka/connect-log4j.properties to :

log4j.rootLogger=INFO, file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=/var/log/kafka/connect.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=5
log4j.appender.file.append=true

For more details regarding log4j you can refer to this article. Also, @Robin Moffatt has written a very nice article for logging REST HTTP messages of Kafka Connect, to a separate file.

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156