On Windows mmap'd files can be automatically extended on open, but this is not supported on UNIX-like systems:
https://docs.python.org/3.7/library/mmap.html
I have an existing set of classes that provide an append-only journal in Python using mmap, but the file size is fixed to 64 MB:
https://github.com/cloudwall/serenity/blob/master/src/serenity/tickstore/journal.py
I'd like to revise
def _check_space(self, add_length: int):
if self.mm.get_pos() + add_length >= self.max_size:
raise NoSpaceException()
to automatically extend the underlying file and add another 64GB of space. In outline I think I know how to do this:
- Unmap the file
- seek to the end
- append 64MB of zeroes
- re-map the file
but am very wary that I'll corrupt a journal on restart if I don't execute this properly. (Note I have live-running services in Kubernetes that are collecting data 24x7.) Is this the right approach? Can you safely extend the file under a mmap region while running?
