0

I have a script that detects files on close_write and runs an ~5 minute process on them. These files are written to the directory in batches of up to 100. The issue is that inotifywait only detects the first file in the batch and does not process the subsequent files unless they are removed from the directory by hand and put back. Here is my script:

#!/bin/bash

inotifywait -r -e close_write -e moved_to --format "%f" $TARGET -m | while read file
    do
        if [[ "$file" =~ .*mp4$ ]]; then
            echo "Detected $file"
            /usr/bin/python3 LongRunningProgram.py -i $TARGET/$file -o $PROCESSED -u $UPLOADPATH -c $C
        fi
    done

it is maintained by a systemctl service written like so:

[Unit]
Description=Description
After=network.target

[Service]
Type=idle
user=pi
WorkingDirectory=/home/pi
ExecStart=/bin/bash /home/pi/notify.sh OutPath C
Restart=on-failure

[Install]
WantedBy=multi-user.target

I am confused as to why it only seems to recognize the first file but not subsequent files when run like this, however if I replace the long running program with sleep 300 it seems to work fine.

Yllier123
  • 66
  • 14
  • 2
    It's not related to this problem, but please remember to always quote your variables, in case the filename contains whitespace. – Barmar May 02 '22 at 16:23
  • I can't think of any reason why it wouldn't recognize the other file. But if it takes it 5 minutes to process each file, you'll have a horrible backlog. It will take over 8 minutes to process the files in a 100-file batch. – Barmar May 02 '22 at 16:27
  • @Barmar there isn't any way I can think of to make the process any faster. Would there perhaps be a way to read all changed filenames at once and then pass all of them to my long running script? It is capable of accepting multiple input files as an argument. – Yllier123 May 02 '22 at 16:35
  • How about running the process in the background, so you don't want for each run before processing the next file. – Barmar May 02 '22 at 16:37
  • @Barmar I had considered that however that would quickly overload the server that these files are being uploaded to. – Yllier123 May 02 '22 at 16:37
  • Your loop could add the filename to an array, and when the array reaches are certain size it can run the script with the array as the arguments, then clear the array in preparation for the next batch. – Barmar May 02 '22 at 16:41
  • @Barmar it would be ideal to let inotifywait work as intended. Is there a recommended way to debug this? Is there maybe a timeout on the pipe read? – Yllier123 May 02 '22 at 17:08
  • No, there are no timeouts on pipes. And even if there were, you'd get the same timeout with `sleep 300` in place of running the script. As I said at the beginning, I can't see any reason why you're losing files. – Barmar May 02 '22 at 17:10

1 Answers1

0

My fault for neglecting to explain what the "long running process" was doing. The python script that is called as a result of inotifywait spins up an ffmpeg process to do work on the MP4 files that were written to the directory. There is a bug with ffmpeg when using it in a while loop. I followed This answers solution to use the -nostdin flag and that appears to have solved the issue. Hopefully this answer helps someone else with this problem :)

Yllier123
  • 66
  • 14