Why is "test -e" not working as it should in my bash script?

Question

I'm writing a short bash script to find if a file exists in a directory or not. Here is the script:

#!/bin/bash

> oldFiles.txt

files=$(grep ' jane ' ../data/list.txt | cut -d ' ' -f 3)

for f in $files; do

f=~$f

if [ -e $f ]; then

  echo $f Found

  echo $f>>oldFiles.txt

else
  echo $f Not found

fi

done

Three relevant file names are found by the grep and searched over in the for loop. It appears, however, that every time my script runs the if command, it comes back False regardless of whether the file exists or not. Please see the results of the script, followed by test -e for each of the individual files, followed by a list of the files in the relevant directory. Also posted below is the referenced list.txt file.

001 jane /data/jane_profile_07272018.doc
002 kwood /data/otherfile.doc
003 pchow /data/pfile.doc
004 janez /data/zfile.doc
005 jane /data/jane_pic_07282018.jpg
006 kwood /data/kwood.jpg
007 pchow /data/pchow.jpg
008 jane /data/jane_contact_07292018.csv
009 kwood /data/kwfile.csv
010 pchow /data/p2file.csv

I know what I'm missing has to be simple, but what is it?

Answer 1

Here's a refactoring which hopefully fixes most of the errors, with inline comments.

#!/bin/bash

# No need for this when you overwrite the file anyway
#> oldFiles.txt

# Don't capture a variable when you only need the result once
# POSIX calls this grep -F
grep -F ' jane ' ../data/list.txt |

# Let read -r do the splitting
#cut -d ' ' -f 3)

# Prefer while read over for
while read -r x y f; do
    # XXX what's this for?
    #f=~$f

    # Fix quoting
    if [ -e "$f" ]; then
        # Diagnostics to stderr
        echo "$0: $f Found" >&2
        # Redirect only once, below
        echo "$f"
    else
        # stderr
        echo "$0: $f Not found" >&2
    fi
# Redirect only once; overwrite
done >oldFiles.txt

If you want to look in your home directory and $f doesn't already have a leading slash, I guess the mystery line should be

    f=~/"$f"

If the file names contain a tilde already (as suggested by your screen shot) the shell will not expand that inside double quotes. You can fix this by manually expanding any leading tilde:

    case $f in
        '~/'*) f=~/"${f#~/}";;
    esac

The parameter expansion $[f#pattern} produces the value of $f with any prefix matching the glob expression pattern trimmed off from the beginning. (There's also ${f%pattern} for trimming from the end, and a good number of other convenient string manipulation facilities.)

If you want /data/stuff to search for a file $HOME/data/stuff or ./data/stuff then you have to add the leading component yourself. The path /data refers to a path directly in the root directory. Perhaps see also Difference between ./ and ~/ which has an answer by yours truly about this.

Answer 2

Don't rely on tilde expansions in your script. In general, in your interactive shell, you can use ~ instead of $HOME since it's easier to type. bash has many other useful tilde expansions, such as ~foo (which expands to the home account of user foo), ~-, ~+, etc, but best practice is to only use these interactively. In your script, just use $HOME instead of ~.

I'd love to give a definitive answer, and the documentation states that no tilde expansion is done in posix mode, but I find that bash will sometimes do tilde expansions when I don't expect it. As a result of that missed expectation, I follow the general practice of simply never using ~ in scripts.

In this case, however, the problem is that bash is doing tilde expansion before variable expansion, and since there's no user named $f (note, that's the literal string $f, not the expansion of the variable f), the tilde prefix ~$f is unchanged by tilde expansion. Then variable expansion is applied and expands the $f, but the ~ remains unchanged. For example:

$ cat a.sh
#!/bin/sh

a=williamp
echo ~williamp
echo ~$a
$ ./a.sh
/Users/williamp
~williamp