xargs split at newlines not spaces

Question

here is my problem in short

$ echo 'for i in $@; do echo arg: $i; done; echo DONE' > /tmp/test.sh
$ echo "ac\nbc\ncc\n" | xargs bash /tmp/test.sh 
arg: ac
arg: bc
arg: cc
DONE

Which is what i expect, but

$ echo "ac s\nbc s\ncc s\n" | xargs -d \n bash /tmp/test.sh
arg: ac
arg: s
arg: bc
arg: s
arg: cc
arg: s
DONE

Shouldn't the output be?

arg: ac s
arg: bc s
arg: cc s
DONE

How do I get the 2nd output with xargs?

Answer 1

Try:

printf %b 'ac s\nbc s\ncc s\n' | xargs -d '\n' bash /tmp/test.sh

You neglected to quote the \n passed to -d, which means that just n rather than \n was passed to xargs as the delimiter - the shell "ate" the \ (when the shell parses an unquoted string, \ functions as an escape character; if an ordinary character follows the \ - n in this case - only that ordinary character is used).

Also heed @glenn jackman's advice to double-quote the $@ inside the script (or omit the in "$@" part altogether).

Also: xargs -d is a GNU extension, which, for instance, won't work on FreeBSD/macOS. To make it work there, see @glenn jackman's xargs -0-based solution.

---

Note that I'm using printf rather than echo to ensure that the \n instances in the string are interpreted as newlines in all Bourne-like shells:
In bash and ksh^[1], echo defaults to NOT interpreting \-based escape sequences (you have to use -e to achieve that) - unlike in zsh and strictly POSIX-compliant shells such as dash.
Therefore, printf is the more portable choice.

^{[1] According to the manual, ksh's echo builtin exhibits the same behavior as the host platform's external echo utility; while this may vary across platforms, the Linux and BSD/macOS implementations do not interpret \ escape sequences by default.}

Answer 2

On Mac OSX

For simple cases that have a known number of args, tell xargs how many args to send to each command. For example

$ printf "1\n2\n3" | xargs -n1 echo "#"
# 1
# 2
# 3

When your input args are complex, and newline terminated, a better method is:

$ printf "1\n2 3\n4 5 6" | xargs -L1 echo  "#"
# 1
# 2 3
# 4 5 6

There is a problem here, can you see it? What if our input line contains a single quote:

$ printf "1\n2 3\n4 '5 6" | xargs -L1 echo  "#"
# 1
# 2 3
xargs: unterminated quote

Single quotes and other quoting characters have special meaning to xargs unless you use the -0 flag. But -0 and -L1 are not compatible, so that leaves us with:

$ printf "1\n2 3\n4 '5 6" | tr '\n' '\0' | xargs -0 -I {} echo "#" {}
# 1
# 2 3
# 4 '5 6

If you brew install findutils we can do a little better:

$ printf "1\n2 3\n4 '5 6" | gxargs -d\\n -i echo "#" {}
# 1
# 2 3
# 4 '5 6

But wait, maybe using xargs is just a bad tool for this one. What if we use the shell builtins instead:

$ printf "1\n2 3\n4 '5 6\n" | while read -r; do echo "# $REPLY"; done
# 1
# 2 3
# 4 '5 6

For some more thoughts about xargs vs while checkout this question.

Answer 3

Your shell script needs to use "$@" not $@

See http://www.gnu.org/software/bash/manual/bashref.html#Special-Parameters

---

I see in the xargs manual on my machine:

xargs reads items from the standard input, delimited by blanks [...] or newlines

(emphasis mine)

Thus:

$ echo $'ac s\nbc s\ncc s\n' | xargs bash /tmp/test.sh  
arg: ac
arg: s
arg: bc
arg: s
arg: cc
arg: s
DONE

$ printf "%s\0" "ac s" "bc s" "cc s" | xargs -0 bash /tmp/test.sh
arg: ac s
arg: bc s
arg: cc s
DONE

With the former, you get the equivalent of

bash /tmp/test.sh ac s bc s cc s

versus using null-separator

bash /tmp/test.sh "ac s" "bc s" "cc s"

You need to be clear about what the delimiter is with xargs when the data contains whitespace.

$ printf "%s\n" "ac s" "bc s" "cc s" | xargs -d $'\n' bash /tmp/test.sh
arg: ac s
arg: bc s
arg: cc s
DONE

$ echo $'ac s\nbc s\ncc s\n' | xargs -d $'\n' bash /tmp/test.sh  
arg: ac s
arg: bc s
arg: cc s
arg:  
DONE

Note the extra arg in the last case, echo already adds a newline, so you don't need an extra one unless you use echo -n

Answer 4

Assuming test.sh is something like

echo arg: "$@"

As already pointed out, you need to escape -d '\n' and it should be noted that this is a GNU extension. It basically is a shortcut to the posix compatible tr '\n' '\0' | xargs -0

But xargs is more versatile and subtle in many cases. It can be used to massage parameters very precisely and can be well illustrated using sh printf. cat -A is used to clearly show the difference.

In a nutshell, xargs splits on whitespace and it treats newlines specially.

To only split on newlines, use -d '\n' or tr '\n' '\0' or use something like sed to escape spaces and tabs.

---

Process invoked once, with three arguments.

printf '%s\n' "ac s" "bc s" "cc s" |
  xargs -d '\n' sh -c 'printf "%s\t" arg: "$@"'';printf "\n"' xargs-example |
  cat -A

arg:^Iac s^Ibc s^Icc s^I$

---

Process invoked once, with 6 arguments.

printf '%s\n' "ac s" "bc s" "cc s" |
   xargs sh -c 'printf "%s\t" arg: "$@"'';printf "\n"' xargs-example |
   cat -A

arg:^Iac^Is^Ibc^Is^Icc^Is^I$

---

Process invoked twice, with 4 and 2 arguments respectively

printf '%s\n' "ac s" "bc s" "cc s" |
    xargs -L2 sh -c 'printf "%s\t" arg: "$@"'';printf "\n"' xargs-example |
    cat -A

arg:^Iac^Is^Ibc^Is^I$
arg:^Icc^Is^I$

---

Process invoked twice, with 2 and 1 arguments respectively.

printf '%s\n' "ac s" "bc s" "cc s" |
     xargs -d '\n' -L2 sh -c 'printf "%s\t" arg: "$@"'';printf "\n"' xargs-example |
     cat -A

arg:^Iac s^Ibc s^I$
arg:^Icc s^I$

---

This may seem a bit complicated but it allows you to use tabs and newlines to control xargs where tabs become argument separators and newlines control which arguments each process is invoked with, ie.

Process invoked twice, with 3 and 1 argument respectively

printf '%s\n' ac$'\t's "bc s" "cc s" |
   sed '
# escape spaces
s@ @\\ @g
   ' |
   xargs -L2 sh -c 'printf "%s\t" arg: "$@"'';printf "\n"' xargs-example |
   cat -A

arg:^Iac^Is^Ibc s^I$
arg:^Icc s^I$