Infinite loop bash script to check for and process file

Question

I am writing a Bash script that searches for a file matching a set pattern in a directory and when found, carries out additional processing.

#!/bin/sh
set -eu

INTERVAL=5
DIRECTORY=/home/methuselah
PATTERN=(MT*xx.txt)

while :
do 
  if [[ -e "$DIRECTORY/$PATTERN" ]]; then
    echo "File has arrived. Starting processing..."
    # Not sure about this!
    for file in "$DIRECTORY/$PATTERN"; do
      echo "foo" > "$file"
    done
  fi
  echo "Waiting for file..."
  sleep $INTERVAL
done

Right now the above script does nothing, even when the file gets dropped in the directory. How can I fix it? And secondly when a file has arrived in the directory, is it possible for me to displa the name?

Update

#!/bin/bash
set -u

INTERVAL=5
DIRECTORY=/home/methuselah
PATTERN=(MT*xx.txt)

while :
do 
  if [[ -e $DIRECTORY/$PATTERN ]]; then
    echo "File has arrived. Starting processing..."
    # How do you print the file name out here?
  fi
  echo "Waiting for file..."
  sleep $INTERVAL
done

And the parameter to `-e` has to be a single filename, you can't use a pattern if it will match multiple files. — Barmar, Oct 17 '19 at 16:51
When you write `PATTERN=(MT*.xx.txt)`, the wildcard will be expanded during the assument. It will set `$PATTERN` to an array of all the matching files. — Barmar, Oct 17 '19 at 16:53
When you use `$PATTERN` later, it will be treated as `${PATTERN[0]}`. — Barmar, Oct 17 '19 at 16:53
You need to quote the pattern during the assignment, and *not* quote it when looking for the files. — Barmar, Oct 17 '19 at 16:54
Thanks @Barmar, how do I display the filename of the matched pattern when it displays — methuselah, Oct 17 '19 at 16:55
You can't use `[[` with `/bin/sh`, you need to use `/usr/bin/bash` — Barmar, Oct 17 '19 at 16:58
Please see updated question, Its working now, it detects the file, I just need it to print out the filename. Is this possible? — methuselah, Oct 17 '19 at 17:00
Did you not see my comment above that you can't use `-e` with multiple filenames? — Barmar, Oct 17 '19 at 17:01
Thanks, if there is more than one file - how would this work? If I need to process them one by one? — methuselah, Oct 17 '19 at 17:02
In general, this isn't the right way to wait for files to be created in the first place. Use a tool like [`incron`](http://inotify.aiken.cz/?section=incron&page=why&lang=en) and you can have your script *invoked* whenever the file changes or creations you care about take place. — Charles Duffy, Oct 17 '19 at 17:03
...moreover, when you use `[ -e whatever ]` to poll, you can trigger when `whatever` is only halfway-written; when you use inotify events as incron does, you can have your script triggered only when the file is *closed* after being fully/completely written. — Charles Duffy, Oct 17 '19 at 17:04
@methuselah No you didn't. Anyway, you can see https://stackoverflow.com/questions/24615535/bash-check-if-file-exists-with-double-bracket-test-and-wildcards/24615684#24615684 — Barmar, Oct 17 '19 at 17:04
@CharlesDuffy The traditional solution to the partially-written file is to write to a different name and then rename it at the end. Rename is atomic. — Barmar, Oct 17 '19 at 17:05
@Barmar, indeed, but I don't see any evidence that the OP here knows the relevant filesystem semantics or even has control over the process doing the writing. — Charles Duffy, Oct 17 '19 at 17:05

Andrew Vickers · Accepted Answer · 2019-10-17T21:06:55.167

3

Be aware of file names that contain spaces! Easiest way I know to handle this is to put each item in an array and then iterate the array:

#!/bin/bash

declare -i interval=5
declare    directory="/home/methuselah"
declare    pattern="MT*xx.txt"    # ${pattern} should not have any spaces.

shopt -s nullglob      # ${pattern} expands to NULL string when no match.
while dirents=( "${directory}"/${pattern} ); do    # Don't quote ${pattern}
  for dirent in "${dirents[@]}"; do    # Quotes keep array elements intact.
    if [ -f "${dirent}" ]; then
      echo "Processing file: '${file}',"
      # Do something to "${file}". Always use quotes.
    fi
  done
  echo "Waiting for file..."
  sleep ${interval}
done

edited Oct 17 '19 at 21:06

answered Oct 17 '19 at 17:08

Andrew Vickers

2,504
2
10
16

Getting an error at this line `done < <( ls $DIRECTORY/$PATTERN )` re: an unexpected token. Which which Bash interpreter are you using? – methuselah Oct 17 '19 at 17:16
Try `/bin/ls` instead of just `ls`! – F. Hauri - Give Up GitHub Oct 17 '19 at 17:17
1

https://mywiki.wooledge.org/ParsingLs - your entire `ls` parsing while loop can be replaced with `files=("$DIRECTORY"/$PATTERN)` – jordanm Oct 17 '19 at 17:18
@jordanm can you post an example. Thanks – methuselah Oct 17 '19 at 17:20
jordanm: Good call! Edited my code as it also gets rid of the subshell redirection problem. – Andrew Vickers Oct 17 '19 at 17:27
You could `while file=($DIRECTORY/$PATTERN) ;[ -f *$file* ];do ...` Instead of testing `${#files[@]}` (`if` and `for`)! – F. Hauri - Give Up GitHub Oct 17 '19 at 17:38
@AndrewVickers thanks, this works but there seems to be a bug at this line `echo "Processing file: '${file}',"`. The ${file} is set to `/home/methuselah/MT*xx.txt` even when there is no match. – methuselah Oct 17 '19 at 17:39
@F.Hauri, ...eh? `[ -f *$file* ]`'s behavior is almost entirely unpredictable; you don't know how many separate words `*$file*` will expand to, so you don't know if the result will be valid `test` syntax or not. – Charles Duffy Oct 17 '19 at 17:44
methuselah: What is your bash version (`bash --version`)? – Andrew Vickers Oct 17 '19 at 17:44
@AndrewVickers, consider `shopt -s nullglob` so the pattern doesn't expand to itself when no matches exist; this should address the issue @methuselah just described above. – Charles Duffy Oct 17 '19 at 17:45
@AndrewVickers it is 4.1.2 – methuselah Oct 17 '19 at 17:45
@AndrewVickers, btw, consider the prior suggestion to make it `"$DIRECTORY"/$PATTERN` (with the directory name's expansion quoted) seconded -- if you don't quote the directory name, it'll be split into multiple words should it contain spaces, and have each word treated as a separate glob expression. – Charles Duffy Oct 17 '19 at 17:46
(Also, using all-caps variable names is bad form -- such names are used for variables that modify behavior of the shell or POSIX-defined operating system components, whereas names with at least one lower-case character are guaranteed not to have unintended side effects on behavior of POSIX-defined tools; see the spec @ https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html, keeping in mind that setting a shell variable modifies any like-named environment variable). – Charles Duffy Oct 17 '19 at 17:47
Modifying code as follows: 1) Use `shopt -s nullglob` 2) Quote `$DIRECTORY` 3) Lower-case all variables 4) use `${variable}` syntax to explicitly delimit variable names – Andrew Vickers Oct 17 '19 at 17:56
I also just realized that there is no need to explicitly test for an non-empty array. This is implicit in the `for file in "${files[@]}"` loop. – Andrew Vickers Oct 17 '19 at 18:03
One more minor optimization: We can assume that assigning an array from the glob will never fail (or if it does, then something catastrophic has happened), so we move the assignment into the while statement. Thanks to F. Hauri whose earlier comment triggered this. – Andrew Vickers Oct 17 '19 at 18:29
@CharlesDuffy Typo: asterisk is behind double quote on my keyboard! `while file=($DIRECTORY/$PATTERN) ;[ -f "$file" ];do ...` will work! – F. Hauri - Give Up GitHub Oct 17 '19 at 19:53
1

@F.Hauri. `$DIRECTORY/$PATTERN` could return more than one filename. You are right to enclose it in parentheses and thereby use array assignment, but you then treat `$file` as a scalar variable. Bash is forgiving here, and gives you the first element of the array, so your code will not fail, but when the pattern matches more than one file, you only process the first one. My version uses the plural `$files` for readability and iterates over that array, thereby processing all files. – Andrew Vickers Oct 17 '19 at 20:57
1

It is probably even better to use the variable name `$dirents` for the array and `$dirent` for elements of the array (or similar) to emphasize the fact that they are directory entries, and might not be files. They could be links, sub-directories, sockets, etc. If you are specifically looking for files, test each `$dirent`. – Andrew Vickers Oct 17 '19 at 20:59
@AndrewVickers Yes, but `while file=(path/file*.dat); [ -f "$file" ];do ..` will process all files, while `$file` is a file (and not just the pattern). Try It, It's fun, you could wipe `if ... for ... done ... fi` and replace by 1 loop. – F. Hauri - Give Up GitHub Oct 18 '19 at 06:08
In current version of your post `while var=(blah) ;do ...` will work like `while :;do var=(blah) ...`! – F. Hauri - Give Up GitHub Oct 18 '19 at 06:14
And yes, I create an *array* and test a *scalar*. But under bash `$varname` is same than `${varname[0]}`. And as array is **redefined** on each loop, `${varname[0]}` will point to the first file of array, or to the pattern if there is no file corresponding to pattern, so test `[ -f $varname ]` will fail. – F. Hauri - Give Up GitHub Oct 18 '19 at 06:16
So instead of `while.. if ... for ...` you will have: `while :;do while file=($pattern); [ -f "$file" ] ;do ... ;done;done`. – F. Hauri - Give Up GitHub Oct 18 '19 at 06:22

Infinite loop bash script to check for and process file

1 Answers1