rm fails to delete files by wildcard from a script, but works from a shell prompt

Question

I've run into a really silly problem with a Linux shell script. I want to delete all files with the extension ".bz2" in a directory. In the script I call

rm "$archivedir/*.bz2"

where $archivedir is a directory path. Should be pretty simple, shouldn't it? Somehow, it manages to fail with this error:

rm: cannot remove `/var/archives/monthly/April/*.bz2': No such file or directory

But there is a file in that directory called test.bz2 and if I change my script to

echo rm "$archivedir/*.bz2"

and copy/paste the output of that line into a terminal window the file is removed successfully. What am I doing wrong?

What's the correct way of doing it if a variable contains the wildcard? Say `archivedir="/var/foo/*.bz2"`. — mandrake, Aug 21 '12 at 07:46

score 135 · Accepted Answer · edited Dec 16 '16 at 07:19

TL;DR

Quote only the variable, not the whole expected path with the wildcard

rm "$archivedir"/*.bz2

Explanation

In Unix, programs generally do not interpret wildcards themselves. The shell interprets unquoted wildcards, and replaces each wildcard argument with a list of matching file names. if $archivedir might contain spaces, then rm $archivedir/*.bz2 might not do what you
You can disable this process by quoting the wildcard character, using double or single quotes, or a backslash before it. However, that's not what you want here - you do want the wildcard expanded to the list of files that it matches.
Be careful about writing rm $archivedir/*.bz2 (without quotes). The word splitting (i.e., breaking the command line up into arguments) happens after $archivedir is substituted. So if $archivedir contains spaces, then you'll get extra arguments that you weren't intending. Say archivedir is /var/archives/monthly/April to June. Then you'll get the equivalent of writing rm /var/archives/monthly/April to June/*.bz2, which tries to delete the files "/var/archives/monthly/April", "to", and all files matching "June/*.bz2", which isn't what you want.

The correct solution is to write:

rm "$archivedir"/*.bz2

Wow, I had no idea the shell does the wildcard operations instead of the actual filesystem programs. This is good to know! — chaz, Apr 26 '13 at 18:07
Interesting explanation , but it does not explain why it works when executed directly at prompt — Emmanuel Devaux, Feb 11 '14 at 09:52
Running "sudo rm /var/lib/OnlyRootCanList/*" will not work when the current user is not allowed to list the files in that directory. In such cases a find command would work better. such as `find . -name "*.jpg" | xargs rm ` — Ray Foss, May 26 '15 at 16:04

score 12 · Answer 2 · answered Apr 26 '09 at 06:18

Your original line

rm "$archivedir/*.bz2"

Can be re-written as

rm "$archivedir"/*.bz2

to achieve the same effect. The wildcard expansion is not taking place properly in your existing setup. By shifting the double-quote to the "front" of the file path (which is legitimate) you avoid this.

score 9 · Answer 3 · edited Apr 26 '09 at 13:27

Just to expand on this a bit, bash has fairly complicated rules for dealing with metacharacters in quotes. In general

almost nothing is interpreted in single-quotes:

 echo '$foo/*.c'                  => $foo/*.c
 echo '\\*'                       => \\*

shell substitution is done inside double quotes, but file metacharacters aren't expanded:
```
 FOO=hello; echo "$foo/*.c"       => hello/*.c
```
everything inside backquotes is passed to the subshell which interprets them. A shell variable that is not exported doesn't get defined in the subshell. So, the first command echoes blank, but the second and third echo "bye":
```
BAR=bye echo `echo $BAR`
BAR=bye; echo `echo $BAR`
export BAR=bye; echo `echo $BAR`
```

(And getting this to print the way you want it in SO ~~takes several tries~~ is apparently impossible...)

score 4 · Answer 4 · answered Apr 26 '09 at 03:45

4

The quotes are causing the string to be interpreted as a string literal, try removing them.

answered Apr 26 '09 at 03:45

Brian Gianforcaro

26,564
11
58
77

score 0 · Answer 5 · answered Sep 09 '11 at 13:13

0

I've seen similar errors when calling a shell script like ./shell_script.sh from another shell script. This can be fixed by invoking it as sh shell_script.sh

answered Sep 09 '11 at 13:13

Christo

161
1
5

This is just cargo cult programming. There is no reason that the wrong shell would cause this particular error. -1 – tripleee Nov 25 '14 at 04:54

score -1 · Answer 6 · answered Dec 26 '17 at 11:13

-1

Why not just rm -rf */*.bz2? Works for me on OSX.

answered Dec 26 '17 at 11:13

holms

9,112
14
65
95

With ZSH it works, with SH it doesn't – Snigdhajyoti Jan 13 '23 at 12:11

rm fails to delete files by wildcard from a script, but works from a shell prompt

6 Answers6

Linked