2

$* is equivalent to $1 $2 $3... - split on all spaces.

"$*" is equivalent to "$1 $2 $3..." - no splitting here.

"$@" is equivalent to "$1" "$2" "$3"... - split on arguments (every argument is quoted individually).

How to quote $(command) so that it treats output lines of the command in the same way "$@" treats arguments?

The problem I want to solve:

I have a backup function that takes files by arguments and backups each of them (e.g.: backup file1 file_2 "file 3"). I want to quickly backup files that are returned by another command.

mycmd returns three files (one per line): file1, file_2 and file 3 (containing a space). If I ran the following command:

backup $(mycmd)

it would be equivalent to running backup file1 file_2 file 3 and it would result in an error because of non-existing files file and 3. However running it this way:

backup "$(mycmd)"

is equivalent to run:

backup "file1
file_2
file 3"

None of them is good enough.

How can I use command substitution to get a call equivalent to: backup "file1" "file_2" "file 3"?

Currently my only workaround is:

while read line; do backup "$line"; done < <(mycmd)
Peter C
  • 73
  • 5
  • 2
    You don't want to parse the output of `ls` (see http://mywiki.wooledge.org/ParsingLs); you should use globs instead of the command substitution, i.e., instead of `$(ls -1 *)` just `*`. So, the answer is "you can't", but also "you shouldn't use command substitution in the first place". – Benjamin W. Jul 27 '17 at 19:57
  • @BenjaminW.: I have already put it in my question. Please read before commenting :-) – Peter C Jul 27 '17 at 20:03
  • 1
    `while read line; do backup "$line"; done` doesn't work adequately. Your code will misbehave with filenames that end with whitespace, filenames that contain literal backslashes, and filenames that contain literal newlines. – Charles Duffy Jul 27 '17 at 20:04
  • 1
    `for file in *; do backup "$file"; done` will do the proper thing and is completely POSIX. – Petr Skocik Jul 27 '17 at 20:06
  • @PSkocik: Backuping * was only a simplification. In the general I need to quote $(command) in general - for any command. – Peter C Jul 27 '17 at 20:10
  • `$(command)` is the wrong tool if you want something safe for the general case. (And by the way, there's an actual builtin command named `command`, making it a questionable choice of placeholders). – Charles Duffy Jul 27 '17 at 20:12
  • How about posting the code for your `backup` script so that we can suggest improvements? As it is, you're looking for a file delimiter that is not a newline, for example what `find`'s `-print0` option provides. We can't help you with your code unless you include your code. – ghoti Jul 27 '17 at 20:13
  • ...if you want to see examples of tools that are written to cover the general case, by the way, you might look at archivers such as (the GNU implementation of) `tar`, or `pax` -- you'll note that they support NUL delimiters when a stream of filenames needs to be passed around. `find -print0`, `pax -0`, `xargs -0`, etc. exist for a reason. – Charles Duffy Jul 27 '17 at 20:14
  • I have edited the question and removed oversimplification so that it is no longer confusing. (It confused BenjaminW. and PSkocik). – Peter C Jul 27 '17 at 20:19

2 Answers2

6

Lists of filenames (or command-line parameters) cannot safely be passed in line-oriented format without escaping.

This is because a command substitution evaluates to a C string. A C string can contain any character other than NUL.

Your intended use case is to generate a list of filenames. An individual filename can also contain any character other than NUL (which is to say: filenames on popular operating systems are allowed to contain literal newlines).

This is true for other command-line parameters as well: foo$'\n'bar is completely valid as an argument-list element.

It is thus literally impossible (without use of an agreed-upon escaping mechanism or higher-level format which one tool knows how to generate and the other knows how to parse) to represent arbitrary filenames in the output from a command substitution.

Safely processing a list of filenames as a string

If you want a stream to safely contain an arbitrary list of filenames, it should be NUL-delimited. This is the format produced by find -print0, for instance, or by the simple command printf '%s\0' *.

However, you can't read this into a shell variable, because (again) a shell variable can't contain the NUL character literal. What you can do is read it into an array:

files=( )
while IFS= read -r -d '' filename; do
  files+=( "$filename" )
done < <(find . -name '*.txt' -print0 )

and then expand that array:

backup "${files[@]}"

Processing line-oriented content (known not to contain newline literals) safely

The above being said, you can read a series of lines into an array, and expand that array (but it isn't safe for the case here, where data is arbitrary filenames):

# readarray is a bash 4.0 builtin
readarray -t lines <(printf '%s\n' "first line" "second line" "third line")
printf 'Was passed argument: <%s>\n' "${lines[@]}"

will properly emit:

Was passed argument: <first line>
Was passed argument: <second line>
Was passed argument: <third line>
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Nice answer Charles! Would `xargs` be a bad thing to use in this situation? Maybe for example ( `find . -maxdepth 1 -type f -print0 | xargs -0 -I{} echo "{}"` )? – l'L'l Jul 27 '17 at 20:16
  • 1
    My hesitance to recommend `xargs` comes from how easy it is to use badly. `-0` helps its safety, but I'd argue that `-I` *hurts* it (given the number of caveats in the definition of that argument's behavior). That said, if `-I` is removed, I agree that `xargs -0` is safe. – Charles Duffy Jul 27 '17 at 20:18
1

for file in *; do backup "$file"; done will do the proper thing and is completely POSIX.

To answer your question, you can use standard IFS-splitting on the outputted strings. IFS is normally ' '$'\t'$'\n'. Perhaps splitting on tabs or newlines alone would solve your problem. Alternatively, you can try splitting on a highly unlikely character such as the vertical tab:

#ensure the outputed items are separated by the char we'll be splitting on
output=$(printf 'a b\vb\vc d') 
set -f #disable glob expansion
IFS=$'\v' 
printf "'%s' " $output; printf '\n'

The above prints 'a b' 'b' 'c d'.

Petr Skocik
  • 58,047
  • 6
  • 95
  • 142
  • 1
    If you're going to suggest string-splitting, you'll need to advise the OP to disable glob expansion to make that safe in the presence of glob-like data elements. – Charles Duffy Jul 27 '17 at 20:20