2

I need to set up a Linux RAMDisk to implement a queue.

The basic idea is there will be processes writing to the RAMDisk. At the same time there are processes that watch for new files and read and delete them as they come.

In the situation that writing is faster than reading and the RAMDisk fills up, the writing should block until there is enough space in the RAMDisk to store the new file that is coming in.

One option would be to use tmpfs, which 'overflows' to disk when the ram assigned to the filesystem is all used - however, the whole system would slow down due to the slow HDD IO involved in that - and in my high-performance system that would be unacceptable.

What I would ideally want is a filesystem that automatically blocks writes when RAMDisk is full until enough space is available.

Note I don't have control over the writing processes, so they will not have the ability to query free space and wait - they are just dumb processes that simply write all the time.

Any solution?

Juan Carlos Petruzza
  • 21
  • 1

1 Answers1

1

There is no simple solution to your problem; a full disk is an error, and needs to be signalled as such. Without the ability to modify the producers, you can't do what you want to do. There is a way to modify the code without modifying it, as it were, involving LD_PRELOAD, but honestly it's not a solution I'd ever want to invite anyone to implement. The correct answer is to design your system properly in the first place, and not to accept "no control" as an excuse for a poor design.

Anyway, given that you're obviously not worried about reliably processing all the jobs (otherwise you wouldn't be trying to implement a queue in a ramdisk), why not just have the producers drop the job if the queue is full?

womble
  • 96,255
  • 29
  • 175
  • 230
  • The 'producers' are external machines sending data to an SMB share. That share will be mounted on (ideally) such mentioned special ramdisk. So I guess I could modify samba code to check for space before writing, or intercept the write system call (w/LD_PRELOAD or other tricks) and do the checking. But I really want to avoid that. On the other side, reliability is handled on a "jobs expected" vs "jobs processed" basis - We do a delta at the end and re-request those not processed. Problem is machines sending by SMB may halt if receive too many full disk errors - we don't control them at all. – Juan Carlos Petruzza Aug 11 '11 at 00:08