0

A single appender writes messages into Chronicle Queue inside Docker without any tailer). The VmRSS keeps growing until daily rolling. Is it expected?

**The machine has plenty of memory and without swap space

tt777
  • 1
  • Can you provide more details please? Growing is a pretty vague term. Growing to gigabytes? Or 1 byte per day? – Dmitry Pisklov Apr 17 '20 at 16:53
  • It can grow to 80 gigabytes at the time before daily rolling. Depends how much data we wrote on that date. The number matches to the size of cq4 file – tt777 Apr 18 '20 at 05:41
  • Are you sure you mean VmRSS and not virtual memory? Virtual memory of the process includes memory mappings, and chronicle uses memory-mapped files, hence the OS would show the whole file length as a mapping although in reality it only keeps a few blocks of 64Mb in resident memory, the rest is just virtual addressing. – Dmitry Pisklov Apr 20 '20 at 10:52
  • Yes, it's VmRSS from /proc/[pid]/status not the VmSize. We are using 6GB block size. If I grep cq4 /proc/[pid]/maps, there are many 7.5GB regions – tt777 Apr 20 '20 at 14:36
  • Ah well if you are using such a huge block size it may well be possible. Basically we will keep a handle to at least 3 blocks: (1) index2index (2) index (3) current writing block In addition, the blocks are lazily released, so there would be potentially a few dangling waiting to be released, especially if you write a lot. Finally on unrelated note - if you want to share data between docker containers, you must use --ipc=host – Dmitry Pisklov Apr 20 '20 at 15:04
  • Is the lazily release triggered by the kernel? Thanks for reminding the --ipc=host argument – tt777 Apr 21 '20 at 15:27
  • It's triggered by the process itself (there's a daemon thread releasing/cleaning up blocks), but the speed of it is limited by your disk speed (and to some extent by your processing power as flushing uses some CPU at the kernel level, too). – Dmitry Pisklov Apr 21 '20 at 20:55
  • If I just want to share between 2 containers, is it good enough for setting --ipc=shareable instead of --ipc=host? Because we had security concerns for setting --ipc=host – tt777 Apr 23 '20 at 16:38
  • You can try but we had issues with --ipc=shareable, YMMV – Dmitry Pisklov Apr 24 '20 at 17:11

0 Answers0