3

The zookeeper transactional logs grow to be very big (Serveral Gigabytes ! ) and there is always one or -at the maximum- two transactional log files in each machine of the cluster knowing that there could be multiple snapshots ! .. Introducing autopurge.purgeInterval,and autopurge.snapRetainCount helped with snapshots but it doesn't have any affect on the transactional logs since the number of those log files are always less than the snapRetainCount !. Is there a way to limit the size of those transactional logs ? and shouldn't the transactional logs be divided into multiple files instead of one big file. Note: I'm using zookeeper-3.4.6

user3580691
  • 88
  • 8
  • I'm using zookeeper with clickhouse clusters. Same problem occurred for me: log files and snapshot files are very large. I just introduced autopurge.purgeInterval,and autopurge.snapRetainCount. Hope it works. I didn't think we could limit the size of logs. – linrl3 Aug 11 '19 at 05:55

1 Answers1

1

If you are using Apache Storm, then you can write a cron job that clears the transaction logs when it reaches a specified/limit size according to your capacity.

I have also worked with storm and faced issues when the zookeeper process used to go down primarily because of large txlog size.

And i wrote a cron job that used to clear large txlogs

Since you will have snapshots created, your topology will be spun back with the same configuration and supervisors will get assigned in case zookeeper restarts

anmol bhatia
  • 11
  • 2