0

I am designing a filesystem based queue and it impacts performance to scan the queue to compute its size when it is very large. I was looking for a way to count the task completions from many different processes. One idea I had was to create a "completions" file that each process opens in append mode and contributes one byte. The size of the file would then be the number of completions, which means the OS is effectively managing all the counter locking for me.

I was wondering if there would be a way to do this without writing any bytes at all using sparse files. I could call ftruncate with size = size + 1, but this would very obviously lead to many race conditions. I was wondering if there was a way to write one byte in append mode and have the byte never materialize.

Thanks for your suggestions!

SapphireSun
  • 9,170
  • 11
  • 46
  • 59
  • Why not having a process that keeps track of the queue in memory. Processes can communicate with this process via memory pipes. – Tarik Jul 25 '20 at 05:17
  • That would be an ordinary and good way to do it, but for this project I'm seeing if I can design a system that requires next to zero upkeep by the programmer. It's totally "serverless". – SapphireSun Jul 27 '20 at 02:22
  • Why not opening the file in update rw mode, lock the first 4 bytes and store the count of task completions in there and then unlock. open, lock, read, increment, write, unlock, close file. Performance might be ok unless you want to flush to disk prior to closing the file. – Tarik Jul 27 '20 at 11:20
  • I might try that approach. The problem is that currently I'm processing somewhere between 250-1000 tasks/sec max and my requirement (on bigger hardware so I don't know how I'm actually doing) is 2500 tasks/sec. I might be prematurely optimizing, but I am leery of this file because it will be attacked by all processes compared with the main queue which has more separable processing. – SapphireSun Jul 28 '20 at 02:06
  • You could revisit having a central process. Any process you start the "server" process if it is not already started. The server process can open the file and lock it. – Tarik Jul 28 '20 at 12:00

0 Answers0