optimize xargs argument enumeration

Question

Can this usage of xargs argument enumaration be optimized better? The aim is to inject single argument in the middle of the actual command.

I do:

echo {1..3} | xargs -I{} sh -c 'for i in {};do echo line $i here;done'

or

echo {1..3} | for i in $(xargs -n1);do echo line $i here; done

I get:

line 1 here
line 2 here
line 3 here

which is what I need but I wondered if loop and temporary variable could be avoided?

Your question is really not clear and really looks like an XY problem. What specific use case do you have in mind? There's very likely a much better design than what you're asking for. — gniourf_gniourf, Jan 22 '17 at 14:22
@gniourf_gniourf - just looking for a neat way of arguments usage, passed through a pipe anywhere in subsequent command — Trebor, Jan 22 '17 at 14:35
That's not really a specific use case. To me, this looks like a broken design from the start. And if you're not experimented in shell scripting, there are lots of caveats you're overlooking (most of them coming from the fact that you're clearly mixing code and data). From here on, one possible scenario is that you'll wrongly learn bad techniques that will bite you in the back sooner than you'll expect. — gniourf_gniourf, Jan 22 '17 at 14:39
@gniourf_gniourf - thanks for your valuable comment.Would you mind pointing me to a resource where I could learn appropriate separation of code and data in the context of shell scripting? — Trebor, Jan 22 '17 at 15:03
@Trebor: I don't really know of any good resources that specifically focus on this point. I can only recommend the best resource on Bash I know of: [The BashFAQ](http://mywiki.wooledge.org/BashFAQ) and [The BashGuide](http://mywiki.wooledge.org/BashGuide). This material is known to be accurate, made by possibly the most knowledgeable and active persons, and is up to date with modern techniques (unlike too many sources out there that are either plain wrong or plain outdated, showing antipatterns and obsolete methods all over the place). — gniourf_gniourf, Jan 22 '17 at 15:19

choroba · Answer 1 · 2017-01-22T14:16:43.137

1

You need to separate the input to xargs by newlines:

echo {1..3}$'\n' | xargs -I% echo line % here

For array expansions, you can use printf:

ar=({1..3})
printf '%s\n' "${ar[@]}" | xargs -I% echo line % here

(and if it's just for output, you can use it without xargs:

printf 'line %s here\n' "${ar[@]}"

)

edited Jan 22 '17 at 14:16

answered Jan 22 '17 at 14:05

choroba

231,213
25
204
289

I did not know about the brace expansion handling prefixes and suffixes. Good to know. It might be worth mentioning that this will not work in other scenarios involving other types of expansion (e.g. expanding all elements in an array). – Fred Jan 22 '17 at 14:12
Note that `echo {1..3}$'\n'` expands to `echo 1$'\n' 2$'\n' 3$'\n'`, so that `echo` sees three arguments. Now `echo` prints its arguments _separated by a space,_ so the `2` and `3` will be prefixed with a space, and you'll also get an empty trailing line (since `echo` also adds a newline at the end). This will cause problems if, later, you want to use (GNU) `xargs` more robustly with `-d\\n`. The `printf` version is obviously superior and more idiomatic. – gniourf_gniourf Jan 22 '17 at 14:20

Fred · Answer 2 · 2017-01-22T14:49:16.077

0

Maybe this?

echo {1..3} | tr " " "\n" | xargs -n1 sh -c ' echo "line $0 here"'

The tr replaces the spaces with newlines, so xargs sees three lines. I would not be surprised if there were a better (more efficient) solution, but this one is quite simple.

Please note I have modified my previous answer to remove the use of {}, which was suggested in the comments to eliminate a potential code injection vulnerability.

edited Jan 22 '17 at 14:49

answered Jan 22 '17 at 14:04

Fred

6,590
9
20

1

More idiomatically, use `printf '%s\n' {1..3}`. – gniourf_gniourf Jan 22 '17 at 14:16
Quite right. I will not update my answer, the other one is already better and more complete, while mine, while not the best, is not wrong either. – Fred Jan 22 '17 at 14:25
Note that your use of `{}` _within `sh` code_ is a very bad thing: you're _mixing code with data!_ (which is one of the main sources of problems, and a horrible antipattern you should recognize and get rid of). Indeed, it's very easy to forge data that will produce arbitrary code execution, e.g., `printf '%s\n' 'one; rm -rf some_precious_directory #' | xargs -I{} sh -c 'echo line {} here;'`, you'll get some surprises! you must handle _data_ as _data._ For this: `... | xargs -n1 sh -c ' echo "line $1 here"' sh`. The same mistake is commonly found with `find`'s `-exec sh -c '... {} ...' \;` – gniourf_gniourf Jan 22 '17 at 14:29
@gniourf_gniourf I would like to modify my answer to correct this problem, but I cannot get the code you are suggesting for xargs to work either with "printf" or with the "tr"-based solution ($1 expands to nothing), all I get printed is "line here". Is there something I am doing wrong? – Fred Jan 22 '17 at 14:43
@gniourf_gniourf actually, it works if I replace $1 by $0. Is that OK? – Fred Jan 22 '17 at 14:45
`printf '%s\n' {1..3} | xargs -n1 sh -c 'echo "line $1 here"' sh`. Note the trailing `sh` here, that will be used as the positional parameter `$0`. You could also use: `printf '%s\n' {1..3} | xargs -n1 sh -c 'echo "line $0 here"'` but usually people find it awkward to use `$0`, so it is customary to feed `sh -c` with a dummy `$0`-th parameter, usually `sh` (but it could be anything else). – gniourf_gniourf Jan 22 '17 at 14:45
Ok, got it. I prefer "$0". If I were to use some trailing value, I would probably use something like DUMMY to make it clear that it is just that. – Fred Jan 22 '17 at 14:46

score 0 · Answer 3 · answered Jan 22 '17 at 15:01

There is a not well known feature of GNU sed. You can add the e flag to the s command and then sed executes whatever is in the pattern space and replaces the pattern space with the output if that command.

If you are really only interested in the output of the echo commands, you might try this GNU sed example, which eliminates the temporary variable, the loop (and the xargs as well):

echo {1..3} | sed -r 's/([^ ])+/echo "line \1 here"\n/ge

it fetches one token (i.e. whatever is separated by the spaces)
replaces it with echo "line \1 here"\n command, with \1 replaced by the token
then executes echo
puts the output of the echo command back into pattern space
that means it outputs the result of the three echos

But an even better way to get the desired output is to skip the execution and do the transformation directly in sed, like this:

echo {1..3} | sed -r 's/([^ ])+ ?/line \1 here\n/g'

score 0 · Accepted Answer · answered Jan 22 '17 at 16:03

Try without xargs. For most situations xargs is overkill. Depending on what you really want you can choose a solution like

# Normally you want to avoid for and use while, but here you want the things splitted.
for i in $(echo {1 2 3} );do 
   echo line $i here;
done

# When you want 1 line turned into three, `tr` can help
echo {1..3} | tr " " "\n" | sed 's/.*/line & here/'

# printf will repeat itself when there are parameters left
printf "line %s here\n" $(echo {1..3})

# Using the printf feature you can avoid the echo
printf "line %s here\n"  {1..3}

optimize xargs argument enumeration

4 Answers4