Curl with multiline of JSON

Question

Consider the curl command below, is it possible to allow newline in JSON (without the minify) and execute directly in bash (Mac/Ubuntu)

curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: text/json; charset=utf-8' \
-d \
'
{
    "field1": "test",
    "field2": {
        "foo": "bar"
    }
}'

When I run the command above, seems error occurred at the second { How to fix the above command?

Updated: actually I was able to run the command without issue previously, not sure why problem happen recently.

Can you tell us more about the error? Your example works "as is" on my system. `mymac > bash --version GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15) Copyright (C) 2007 Free Software Foundation, Inc.` — Eric Bolinger, Jan 27 '16 at 21:00
Yup, works for me as well: `GNU bash, version 4.3.42(1)-release` — miken32, Jan 29 '16 at 04:43
Also check out [ANSI C-like string syntax](http://wiki.bash-hackers.org/syntax/quoting): `echo $'here is a newline:\nand here is a tab:\t'` — miken32, Jan 29 '16 at 19:50
`application/json` is the correct media type for JSON data -- see [RFC4627](https://www.ietf.org/rfc/rfc4627.txt) — Pocketsand, Jan 10 '19 at 19:02

Eric Bolinger · Answer 1 · 2019-11-03T21:15:16.523

211

I remembered another way to do this with a "Here Document" as described in the Bash man page and detailed here. The @- means to read the body from STDIN, while << EOF means to pipe the script content until "EOF" as STDIN to curl. This layout may be easier to read than using separate files or the "echo a variable" approach.

curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: application/json; charset=utf-8' \
--data-binary @- << EOF
{
    "field1": "test",
    "field2": {
        "foo": "bar"
    }
}
EOF

NOTE: Use the --trace <outfile> curl option to record exactly what goes over the wire. For some reason, this Here Document approach strips newlines. (Update: Newlines were stripped by curl -d option. Corrected!)

edited Nov 03 '19 at 21:15

answered Apr 21 '16 at 18:47

Eric Bolinger

2,722
1
13
22

6

This is clean, no extra quoting, no escaping and it works very well. Thanks. – Seth Dec 20 '16 at 17:53
1

Can we use pipes with such a syntax? – Ivan Balashov Mar 08 '18 at 14:31
3

Yes you can pipe the output to another command, although the placement is right in the middle. Add the stdout redirect after the stdin redirect. Example using word count: `-d @- << EOF | wc` – Eric Bolinger May 04 '18 at 15:15
2

It's not Here Document that strips new lines, but `curl -d` option: https://curl.haxx.se/docs/manpage.html#-d. Use `--data-binary` to keep newline and carriage return characters. – dzieciou Jun 28 '19 at 12:30
Can you insert variables into the JSON? I don't think so. – Oliver Pearmain Mar 11 '21 at 15:12
1

You most certainly can insert variables! See what happens when you replace "foo": "bar" with "$SHELL": "$(date)". – Eric Bolinger Mar 14 '21 at 21:07
This works like a charm. And it also supports passing values to the JSON with ${VAR} – Nathan5x Jun 29 '22 at 18:56
Finally, I found a great solution for this! Thank you. This should be the accepted answer. – mljrg Mar 31 '23 at 17:15

Bampfer · Answer 2 · 2016-02-01T13:17:51.217

58

Along the lines of Martin's suggestion of putting the JSON in a variable, you could also put the JSON in a separate file, and then supply the filename to -d using curl's @ syntax:

curl -0 -v -X POST http://www.example.com/api/users \
  -H "Expect:" \
  -H 'Content-Type: text/json; charset=utf-8' \
  -d @myfile.json

The disadvantage is obvious (2 or more files where you used to have one.) But on the plus side, your script could accept a filename or directory argument and you'd never need to edit it, just run it on different JSON files. Whether that's useful depends on what you are trying to accomplish.

edited Feb 01 '16 at 13:17

answered Jan 29 '16 at 22:32

Bampfer

2,120
16
25

1

Note: make sure you've valid json content through https://jsonlint.com/ – vikramvi Feb 20 '18 at 14:10
This approach is clean and easy to debug compared to others. – vikramvi Feb 20 '18 at 14:22
Great solution. Be careful of PATH tho ! – Pierre Ferry Oct 23 '18 at 15:30

score 33 · Answer 3 · answered Oct 17 '17 at 20:58

For some reason, this Here Document approach strips newlines

@eric-bolinger the reason the Heredoc strips newlines is because you need to tell your Heredoc to preserve newlines by quoting the EOF:

curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: text/json; charset=utf-8' \
-d @- <<'EOF'

{
    "field1": "test",
    "field2": {
        "foo": "bar"
    }
}
EOF

Notice the single-ticks surrounding EOF the first time it's defined, but not the second.

score 30 · Answer 4 · answered Jan 18 '16 at 05:49

You should use outer double quotes, and the escape all inner quotes like this:

curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: text/json; charset=utf-8' \
-d \
"
{
    \"field1\": \"test\",
    \"field2\": {
        \"foo\": \"bar\"
    }
}"

score 28 · Answer 5 · answered Jan 18 '16 at 06:00

28

You could assign your json to a var:

json='
{
    "field1": "test",
    "field2": {
        "foo": "bar"
    }
}'

Now you can forward this to curl using stdin:

echo $json | curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: text/json; charset=utf-8' \
-d @-

answered Jan 18 '16 at 06:00

Martin Konecny

57,827
19
139
159

1

Using single quotes to surround the block means that you *cannot* use variables (e.g. `${username}`) in the JSON. – Air Jan 06 '17 at 03:10
1

Yup, but using double quotes means you can't use $ signs in your data. Pick which one is right for you. – Martin Konecny Jan 06 '17 at 06:13

score 10 · Answer 6 · answered Oct 28 '21 at 21:50

10

I think this can be an answer

curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: text/json; charset=utf-8' \
--data-raw '
{
    "field1": "test",
    "field2": {
        "foo": "bar"
    }
}'

answered Oct 28 '21 at 21:50

kangkyu

5,252
3
34
36

I like this because it works well even with indentation and if one uses "" quotes, even variables can be used. – lnksz Aug 05 '22 at 16:25

Curl with multiline of JSON

6 Answers6

Linked