Do we need a second thread to process time-consuming jobs with inotify?

Question

Referenc: How to monitor a folder with all subfolders and files inside?

I need to monitor a directory for any file with extension of *.log. If a new file is created or is moved in, I will process the file. it takes variable time to process each file.

Question> Do I need to create one thread to listen the inotify event and push the new file name into queue and use another thread to process the queue? My concern is that if I don't use separate threads, then the inotify may fail to track changes caused by some large files.

I have simulated the problem with sleep while processing each log file without any multi-thread code. It seems to me that inotify always corrects get all created/moved files in the directory.

Here is my simulation.

Terminate 1: I run the app and listen the inotify event within the main. Whenever processing one log file, I will sleep for 10 second then print the file name to console.

Terminte 2: I will copy multiple files at time T1 to the monitored directory. Before the app finishing processing all previous files, I will move multiple files at time T2. Then again time T3, I will copy multiple files to the directory before the app finishes.

Observation: The app processed all log files as I expected.

Question> Is this expected behavior of inotify or I just run lucky this morning? In other words, will inotify cache unprocessed events in case of simulation as above?

FYI: the code snippet I used:

#define EVENT_SIZE    ( sizeof (struct inotify_event) )
#define BUFFER_LEN  ( 1024 * ( EVENT_SIZE + NAME_MAX + 1) )

wd = inotify_add_watch( fd, root.string().c_str(), IN_CREATE | IN_MOVED_TO );

while ( true )
{
    length = read( fd, buffer, BUFFER_LEN );

    for ( int i = 0; i < length; )
    {
        const inotify_event *ptrEvent = reinterpret_cast<inotify_event *>(&buffer[i]);
        ... // processing the event
        sleep(10); // to simulate the long work
        i += INO_EVENT_SIZE + ptrEvent->len;
    }
}

will the fd returned by inotify_add_watch() work with `select()`? — Richard Hodges, May 11 '16 at 13:35
@RichardHodges, check this one http://stackoverflow.com/questions/37125002/why-perform-selective-read-on-the-inotify-instance-using-select-call — q0987, May 11 '16 at 13:57

score 0 · Answer 1 · answered May 11 '16 at 15:26

Let's do a thought experiment. Clearly, there is some sort of a queue with events for you to read. But let's say you don't read them. If the queue is unbound in size, you will run out of memory, so that's a non-starter. If the queue is bound, you either have to start dropping events at some point or BLOCK operations which would generate said events. The latter is obviously a non-starter - you don't want your fs operations to block because a process with inotify watchers is waiting on some crap and not reading anything.

But most importantly, you very likely don't want to use inotify in the first place. The entire business here looks misdesigned. What you likely want is one directory, right, which gets files renamed into. So that when a file appears, it is safe to read. Then, depending on what this is about, you may want to just get SIGUSR1 or something to trigger dir processing or get a request over a unix socket with exact file name.

The dir in question must only contain unprocessed files.

Do we need a second thread to process time-consuming jobs with inotify?

1 Answers1