0

I have a Flowable that receives filesystem events (e.g: file updated, using inotify), buffers the files while the filesystem edits complete, and then hands these events off for processing.

This downstream processing, which occurs in different threads, can potentially update the files that are updated. So that creates new filesystem events.

fileUpdateFlowable                      <- Flowable<FileEvent>
  .onBackPressureDrop(...)
  .flatMap(fileEvent ->
    Flowable.just(fileEvent)
      .map(fileEvent -> {
        if(fileIsStable) return file;
        else throw new UnstableFileException();
      })
      .retryWhen(f -> {
        f.flatMap(error ->
          if(error instanceof UnstableFileException) return Flowable.timer(delay);
          else throw error;
        )
      })

How can I make the "processing" respect the state at the top of the Flowable? The buffering of files means that I can run into backpressure if:

  • There are too many threads processing downstream.
  • The threads happen to edit a lot of files.
  • The types of file edits happen to create more filesystem events (this depends on what is done to the files).

I'm looking for a way for the processing threads to delay writing while there are a lot of filesystem events backed up.

I looked into running the edits within the same Scheduler as the filesystem event processing, however the way the buffer currently works is to retryWhen with a Flowable.timer - so there is no obvious way to delay this work.

Dan Gravell
  • 7,855
  • 7
  • 41
  • 64
  • 1
    Are the filesystem events only triggered by your processing of files? How many events are triggered by your processing per file? Why do you need to rely on such file modification events? What was the original requirement you tried to solve with this setup? – akarnokd Mar 10 '21 at 17:36
  • There are possibly other sources of filesystem events, but predominantly they are triggered by my processing. – Dan Gravell Mar 10 '21 at 19:24
  • The number of events depends on the nature of the change. Some file changes can result in one event, some can result in more. It can also depend on the platform - inotify might send more changes than the Windows equivalent. Currently I'm using the default buffer size of 128 downstream, but I want to avoid relying on buffers as much as I can, hence trying to slow down the "processing" threads somehow. I can make sure I take unique events, for example, but ultimately coping with only 128 changed files during the buffering period is on the low side, and I also run on resource constrained devices. – Dan Gravell Mar 10 '21 at 19:27
  • I rely on file modification events because I also want to process out of band changes from other software, e.g. file transfer software depositing files in the watched folder. – Dan Gravell Mar 10 '21 at 19:28
  • 1
    I don't think you can apply lossless backpressure to the flow. You could coalesce the change notification per file via PublishProcessor+Latest but if there are more files than the buffer size acting on the changes, you are risking deadlock. – akarnokd Mar 11 '21 at 10:25
  • Thanks. By "lossless" do you mean avoidance of loss of events via backpressure? Ultimately, because events may come externally, I realise lossless is impossible. However, I was hoping that I could somehow at least reduce my own event production to make this less likely. – Dan Gravell Mar 11 '21 at 11:45
  • @akarnokd sorry to bother you but just wondered if there were any thoughts you had about using the existing Flowable/Schedulers/threads to slow down the production of events from my own code (as I said, I realise completely lossless is impossible because the Flowable is hot and some events will be produced outside of my control - I just think I can improve the situation). – Dan Gravell Mar 16 '21 at 11:21
  • 1
    If you can split up the work along operations that trigger individual file events, you can use Flowable of such work and limit the buffering/prefetch on those parts. – akarnokd Mar 16 '21 at 11:44
  • Thanks, I'll look into how I can do that. – Dan Gravell Mar 16 '21 at 12:58

0 Answers0