-1

I have kafka consumer and output which is live stream of data should be captured on hourly basis in a log file. Took help from this link [1]: https://richard-purves.com/2017/11/08/log-rotation-mac-admin-cheats-guide/

Created a log file kafka.log in /var/log and config file for the same named kafka.conf in /etc/newsyslog.conf

Tried running kafka consumer that creates log file under /var/log/kafka.log

python3 consumer.py --kafka-broker "xxx" --kafka-topic "yyy" --kafka-group-id cxel-yellow-vsg-qoe --kafka-ssl-keyfile "key" --kafka-ssl-certfile "certificate --kafka-ssl-cafile /usr/local/etc/openssl@1.1/cert.pem --schemas prod --kafka-security-protocol SSL >> sudo /var/log/kafka.log 2>&1 &

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Madhuvanthi V
  • 21
  • 6
  • 2
    [so] is for programming questions, not questions about using or configuring Unix and its utilities. [unix.se] or [apple.se] would be better places for questions like this. – Barmar Sep 21 '21 at 21:09

1 Answers1

0

kafka consumer [whose] output [is a] live stream of data [and] should be captured on hourly basis in a log file.

This is precisely the use case for Kafka Connect, not some standard Python consumer

Kafka comes with connect-standalone.sh to get you started

You may use the HDFS2 Sink Connector provided by Confluent to write data into (approximate} hourly files (one file per topic partition) using its HourlyPartitioner class, bounded also by a consumer fetch size

You don't need to install HDFS or download Hadoop to use it. In the configuration for a file location, you can provide file:///tmp/kafka-logs, for example

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245