2

My understanding of FUSE's multithreaded read cycle is something like this:

            ....

       .->  read  --.
      /              \
open  --->  read  ----+->  release
      \              /
       `->  read  --'

            ....

i.e., Once a file is open'd, multiple read threads are spawned to read different chunks of the file. Then, when everything that was wanted has been read, there is a final, single release. All these are per ones definition of open, read and release as FUSE operations.

I'm creating an overlay filesystem which converts one file type to another. Clearly, random access without any kind of indexing is a problem; so for the time being, I'm resorting to streaming. That is, in the above model, each read thread would begin the conversion process from the start, until it arrives at the correct bit of converted data to push out into the read buffer.

This is obviously very inefficient! To resolve this, a single file conversion process can start at the open stage and use a mutex and read cursor (i.e., "I've consumed this much, so far") that the reader threads can use to force sequential access. That is, the mutex gets locked by the thread that requests the data from the current cursor position and all other reader threads have to wait until it's their turn.

I don't see why this wouldn't work for streaming a file out. However, if any random/non-sequential access occurs we'll have a deadlock: if the requested offset is beyond or before the current cursor position, the mutex will never unlock for the appropriate reader to be able to reach that point. (Presumably we have no access to FUSE's threads, so to act as a supervisor. Also, I can't get around the problem by forcing the file to be a FIFO, as FUSE doesn't support writing to them.)

Indeed, I would only be able to detect when this happens if the mutex is locked and the cursor is beyond the requested offset (i.e., the "can't rewind" situation). In that case, I can return EDEADLK, but there's no way to detect "fast-forward" requests that can't be satisfied.

At the risk of the slightly philosophical question... What do I do?

Xophmeister
  • 8,884
  • 4
  • 44
  • 87
  • Obviously one solution would be to implement proper random access :P – Xophmeister Oct 20 '15 at 15:15
  • fuse parallelizes multiple fuse requests. So calling getattr() on different files can run in parallel. The vfs inode locking limits this. And fuse does not split up large requests into multiple small requests. I'm not sure multiple read requests to the same file will ever happen in parallel. But if they do happen that means that someone is doing multiple read requests to the file in question and it is unlikely they would be sequential at all. Implement caching for your conversion and handle random access. – Goswin von Brederlow Oct 10 '17 at 12:34

0 Answers0