0

We are implementing a memory consuming long-running server, which use jemalloc inside for alloc/dealloc memory. The server process data batch-by-batch, continuously. With each batch, it allocates memory (via malloc()) and process data inside the batch, after which memory is released (via free()).

The problem is, when new data come in, it has to allocate new memory for processing, which incur lots of page fault because memory pages allocated from the operating system previously are returned back on previous free() (or, purged by madvise)

Is there any way to retain (in process address space) memory which are freed previously so that next allocation can reuse them?

Note that because each allocation/deallocation are not fixed size so a fixed-size memory pool is not suitable here. Some general allocator like jemalloc is needed here.

Is there configuration in jemalloc so that all allocated-and-freed memory retain in process address space for later reuse? It is acceptable to allocate a large chunk (like, 64GB) on server startup and handle allocation/deallocation/fragmentation/defragmentation inside this single piece.

walkerlala
  • 1,599
  • 1
  • 19
  • 32
  • I think you want to use `mallctl` to set `opt.retain` to true. See the manpage: http://jemalloc.net/jemalloc.3.html#mallctl_namespace – Oppen Apr 15 '20 at 14:22
  • @Oppen It seems that `opt.retain` is only for unused page, not dirty page – walkerlala Apr 16 '20 at 02:41
  • Do you mean it only avoids giving back pages that it asked for future use, when no use indeed happen? Have you tried writing a sample program and `strace`ing it to compare the effects? – Oppen Apr 16 '20 at 22:59

0 Answers0