Correct way of quoting command substitution

Question

I have simple bash script which only outputs the filenames that are given to the script as positional arguments:

#!/usr/bin/env bash

for file; do
    echo "$file"
done

Say I have files with spaces (say "f 1" and "f 2"). I can call the script with a wildcard and get the expected output:

$ ./script f*
> f 1
> f 2

But if I use command substitution it doesn't work:

$ ./script $(echo f*)
> f
> 1
> f
> 2

How can I get the quoting right when my command substition outputs multiple filenames with spaces?

Edit: What I ultimatively want is to pass filenames to a script (that is slightly more elaborate than just echoing their names) in a random order, e.g. something like that:

./script $(ls f* | shuf)

Is your example just to demonstrate or are you actually trying to echo a glob? What your asking for isn't directly possible, but there are usually ways to work around the limitation depending on what you're really trying to do. — jordanm, Aug 12 '21 at 14:58
`$(echo f*)` is split into words based on `IFS`; quoting it like `"$(echo f*)"` would convert the output into a single argument. What exactly is it you're trying to do – do you have an array of files somewhere that you want to pass to `script`, or to you pass around a glob? There probably are ways to convert your glob into an array and then use that, but I'd need a little more context. — Benjamin W., Aug 12 '21 at 15:02
I'm mostly asking because the solution to the problem as presented is "just use the glob". — Benjamin W., Aug 12 '21 at 15:03
Command substations produce *a* string, not a series of values. A string containing spaces is in no way equivalent to a list of strings separated by spaces. `shuf` produces, in some sense, a single linefeed-delimited string values, but that value cannot guarantee that each line corresponds to exactly one complete file name, because filenames can contain newlines. — chepner, Aug 12 '21 at 15:40
See http://mywiki.wooledge.org/ParsingLs for some additional information. — chepner, Aug 12 '21 at 15:41

Benjamin W. · Answer 1 · 2021-10-01T09:44:29.863

With GNU shuf and Bash 4.3+:

readarray -d '' files < <(shuf --zero-terminated --echo f*)
./script "${files[@]}"

where the --zero-terminated can handle any filenames, and readarray also uses the null byte as the delimiter.

With older Bash where readarray doesn't support the -d option:

while IFS= read -r -d '' f; do
    files+=("$f")
done < <(shuf --zero-terminated --echo f*)
./script "${files[@]}"

In extreme cases with many files, this might run into command line length limitations; in that case,

shuf --zero-terminated --echo f*

could be replaced by

printf '%s\0' f* | shuf --zero-terminated

_{Hat tip to Socowi for pointing out --echo.}

score 1 · Answer 2 · answered Aug 12 '21 at 15:31

It's very difficult to get this completely correct. A simple attempt would be to use %q specifier to printf, but I believe that is a bashism. You still need to use eval, though. eg:

$ cat a.sh
#!/bin/sh

for x; do echo $((i++)): "$x"; done
$ ./a.sh *
0: a.sh
1: name
with
newlines
2: name with spaces
$ eval ./a.sh $(printf "%q " *)
0: a.sh
1: name
with
newlines
2: name with spaces

score 1 · Answer 3 · answered Aug 12 '21 at 15:43

This feels like an XY Problem. Maybe you should explain the real problem, someone might have a much better solution.

Nonetheless, working with what you posted, I'd say read this page on why you shouldn't try to parse ls as it has relevant points; then I suggest an array.

lst=(f*)
./script "${lst[@]}"

This will still fail if you reparse it as the output of a subshell, though -

./script  $( echo "${lst[@]}" )  #  still  fails same way
./script "$( echo "${lst[@]}" )" # *still* fails same way

Thinking about how we could make it work...

pynexj · Answer 4 · 2021-08-12T16:34:00.357

You can use xargs:

$ ls -l
total 4
-rw-r--r-- 1 root root  0 2021-08-13 00:23 '  file  1'
-rw-r--r-- 1 root root  0 2021-08-13 00:23 '  file  2'
-rw-r--r-- 1 root root  0 2021-08-13 00:23 '  file  3'
-rw-r--r-- 1 root root  0 2021-08-13 00:23 '  file  4'
-rwxr-xr-x 1 root root 35 2021-08-13 00:25  script
$ ./script *file*
  file  1
  file  2
  file  3
  file  4
$ ls *file* | shuf | xargs -d '\n' ./script
  file  4
  file  2
  file  1
  file  3

If your xargs does not support -d:

$ ls *file* | shuf | tr '\n' '\0' | xargs -0 ./script
  file  3
  file  1
  file  4
  file  2

Correct way of quoting command substitution

4 Answers4