xargs: how to have literal double-quotes in replacement?

Question

I have a source file with JSON-Objects one per line like this:

source:

{"_id":"1","name":"one"}
{"_id":"2","name":"two"}
{"_id":"3","name":"three"}

I want to send each line to a

curl -X POST -H "application/json" myURL -d '<REPLACEMENT>'

The double quotes do not make it to curl when I am trying

<source xargs -I % curl -X POST -H "application/json" myURL -d '%'

I tried escaping the quotes in the curl command and later I replaced all double-quotes in the source file with \". I found no version to work.

Another approach to use seq with sed to write each line into a temp file and curl -d @temp did not work out for me.

Is there an elegant solution or do I have to write a script with a loop?

score 9 · Answer 1 · answered Jul 16 '14 at 16:07

9

That's an interesting problem. There must be a better solution (perhaps konsolebox is onto something), but substituting all " with \" would work:

$ echo '"hello"'
"hello"
$ echo '"hello"' | xargs echo
hello
$ echo '"hello"' | sed 's/"/\\"/g' | xargs echo
"hello"

answered Jul 16 '14 at 16:07

ooga

15,423
2
20
21

In fact I need two levels of quotations to be send to the looped curl: the inner " are part of the json syntax and the outer ' limit the value of the -d prameter. I tried with escaping the source but with curl -d '%' I had no luck. Maybe try it again. – tschloss Jul 17 '14 at 06:47

score 5 · Accepted Answer · answered Jul 16 '14 at 17:45

5

GNU Parallel was built specifically to deal with xargs bad handling of special chars:

<source parallel curl -X POST -H "application/json" myURL -d {}

Not only will it quote " correctly, it will quote any string correctly, so it will be interpreted as a single argument by curl.

Added bonus: Your queries will run in parallel - one query per cpu.

answered Jul 16 '14 at 17:45

Ole Tange

31,768
5
86
104

Thx, sounds good! I was lucky: parallel is a known recipe of homebrew - so it is installed - try it later. Thx so far for the hint! – tschloss Jul 17 '14 at 06:51
Ok, "parallel" instead of xargs did help with this change: the H-value quoting had to be escaped too: ` – tschloss Jul 17 '14 at 08:59

score 2 · Answer 3 · answered Jan 30 '18 at 11:38

Should do the trick:

cat source.json | xargs -0 -I {} curl {}

From man xargs:

  -0, --null
         Input items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (ev‐
         ery  character  is  taken  literally).  Disables the end of file string, which is treated like any other argument.  Useful
         when input items might contain white space, quote marks, or backslashes.  The GNU find -print0 option produces input suit‐
         able for this mode.

konsolebox · Answer 4 · 2014-07-16T16:10:54.050

0

Try to use --data-urlencode:

<source xargs -I % curl -X POST -H "application/json" myURL --data-urlencode '%'

The option may be used with other formats. See the manual of curl. You can also try --data-binary.

edited Jul 16 '14 at 16:10

answered Jul 16 '14 at 15:57

konsolebox

72,135
12
99
105

Thanks. Does not help. I need to escape the single quotes around % to \'%\' - but neither in urlendcode nor binary mode the inner double quotes come through. The behaviour of curl when receiving the broken json is even worse. – tschloss Jul 17 '14 at 08:22

xargs: how to have literal double-quotes in replacement?

4 Answers4