JSON requests in C using libcurl

Question

I'm defining a PUT request with a JSON request body using libcurl in C.

This how I'm doing it:

    sprintf(jsonObj, "\"name\" : \"%s\", \"age\" : \"%s\"", name, age);

    struct curl_slist *headers = NULL;
    curl_slist_append(headers, "Accept: application/json");
    curl_slist_append(headers, "Content-Type: application/json");
    curl_slist_append(headers, "charset: utf-8");

    curl_easy_setopt(curl, CURLOPT_URL, url);

    curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers); 
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, jsonObj);
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)&chunk);
    curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcrp/0.1");

    res = curl_easy_perform(curl);

The request body is arriving like this:

    { '"name" : "Pedro", "age" : "22"' }

With { ' at the start and ' } at the end.

--- MORE INFO -----

if I declare this code

    char* jsonObj = "{ \"name\" : \"Pedro\" , \"age\" : \"22\" }"; 

    struct curl_slist *headers = NULL;
    curl_slist_append(headers, "Accept: application/json");
    curl_slist_append(headers, "Content-Type: application/json");
    curl_slist_append(headers, "charset: utf-8");

    curl_easy_setopt(curl, CURLOPT_URL, url);

    curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers); 
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, jsonObj);
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)&chunk);
    curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcrp/0.1");

    res = curl_easy_perform(curl);

the server receives this as the request body:

{ '{ "name" : "Pedro" , "age" : "22" }': '' }

My question is:

Is the libCurl pre-formatting/encoding automatically the Json request?

By the way, does the libCurl have some way of encoding a JSON object?

Thanks so much!

score 25 · Accepted Answer · edited Feb 29 '20 at 12:58

25

The problem may be with the headers. When you are configuring your curl_slist headers I think you should assign the output of curl_slist_append back to headers:

struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Accept: application/json");
headers = curl_slist_append(headers, "Content-Type: application/json");
headers = curl_slist_append(headers, "charset: utf-8");

edited Feb 29 '20 at 12:58

Jens A. Koch

39,862
13
113
141

answered Oct 17 '12 at 16:14

JustinB

1,532
15
18

1

This is the correct solution. Without the headers, the JSON sent is wrapped in a {' ..... ',''} string. – blorkfish Jan 18 '17 at 06:15
2

What is the `"charsets: utf-8"` header? Looks like a mistake. You probably should use `Content-Type: application/json; charset=utf-8` instead. – Paul Nov 08 '18 at 21:56
Just to clarify, `application/json` is _denotes_ the UTF-8 encoding of Unicode. Appending `; charset=UTF-8` to the MIME type is deprecated. Nor is a separate `charset: UTF-8` header wanted (that is not a standard header and will not be recognized by servers). – Charlie Reitzel Mar 22 '23 at 14:09

score 17 · Answer 2 · edited Feb 29 '20 at 13:00

Firstly, let's note a few things. To start with, Daniel Stenberg is correct (as I'd hope he would be, given that he wrote the code): libcurl does not append any data to your code. I can demonstrate this with this sample program, which is like your example but with some additional setup/teardown code. This is all of the code: there is nothing else present here:

#include <curl/curl.h>

int main (int argc, char *argv[]) {
    CURL *curl;
    CURLcode res;

    curl_global_init(CURL_GLOBAL_ALL);
    curl = curl_easy_init();
    if (curl == NULL) {
        return 128;
    }

    char* jsonObj = "{ \"name\" : \"Pedro\" , \"age\" : \"22\" }";

    struct curl_slist *headers = NULL;
    curl_slist_append(headers, "Accept: application/json");
    curl_slist_append(headers, "Content-Type: application/json");
    curl_slist_append(headers, "charset: utf-8");

    curl_easy_setopt(curl, CURLOPT_URL, "http://http2bin.org/put");

    curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, jsonObj);
    curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcrp/0.1");

    res = curl_easy_perform(curl);

    curl_easy_cleanup(curl);
    curl_global_cleanup();
    return res;
}

Packet capturing the request with Wireshark shows that this emits the following HTTP request:

PUT /put HTTP/1.1
Host: http2bin.org
User-Agent: libcrp/0.1
Accept: */*
Content-Length: 35
Content-Type: application/x-www-form-urlencoded

{ "name" : "Pedro" , "age" : "22" }

As you can see, this is exactly the JSON data you asked to send. There is no extra data here, no enclosing braces.

This means that the extra braces are being added either by your server or by some intermediate middlebox. My guess is that your server is adding it because it is forcibly trying to turn any body that is not an application/json body into one by considering the entire body a string.

The reason your server doesn't consider this a JSON body is encapsulated by another answer here: you aren't setting your headers properly. curl_slist_append returns a new struct curl_slist * that you need to assign back into your headers variable. That means you need to change these four lines:

struct curl_slist *headers = NULL;
curl_slist_append(headers, "Accept: application/json");
curl_slist_append(headers, "Content-Type: application/json");
curl_slist_append(headers, "charset: utf-8");

to these four:

struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Accept: application/json");
headers = curl_slist_append(headers, "Content-Type: application/json");
headers = curl_slist_append(headers, "charset: utf-8");

This should convince your server that you are sending JSON data.

In the future, I recommend you get familiar with Wireshark for solving problems like this. It is extremely helpful to see the actual request you sent. Failing that, if you'd rather not, you can use CURLOPT_DEBUGFUNCTION to grab the data curl is sending to validate it.

Thanks for pasting *ALL* of the code. It would be easier to answers question on SO if people wouldn't only paste 1% of an irrelevant part of their code... — 7heo.tk, Jan 18 '17 at 08:25

Rahim Khoja · Answer 3 · 2016-05-02T15:51:13.570

I was having a similar Issue.

I discovered the -libcurl option for the curl command. It helped a lot! Just add it to the end of your working curl command.

In the end it help me create this code:

  CURLcode ret;
  CURL *hnd;
  struct curl_slist *slist1;
  std::string jsonstr = "{\"username\":\"bob\",\"password\":\"12345\"}";

  slist1 = NULL;
  slist1 = curl_slist_append(slist1, "Content-Type: application/json");

  hnd = curl_easy_init();
  curl_easy_setopt(hnd, CURLOPT_URL, "http://u/r/l");
  curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
  curl_easy_setopt(hnd, CURLOPT_POSTFIELDS, jsonstr.c_str());
  curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.38.0");
  curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, slist1);
  curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
  curl_easy_setopt(hnd, CURLOPT_CUSTOMREQUEST, "POST");
  curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);

  ret = curl_easy_perform(hnd);

  curl_easy_cleanup(hnd);
  hnd = NULL;
  curl_slist_free_all(slist1);
  slist1 = NULL;

Express recieves this JSON as:

{ username: 'bob', password: '12345' }

I hope this helps!

I've just discovered `--libcurl` option because this answer. It helps a lot! Thank you! — logoff, May 24 '18 at 15:43

Daniel Stenberg · Answer 4 · 2017-01-18T08:27:56.070

3

libcurl will send exactly the bytes you ask it to send. It has no knowledge of JSON at all.

See @Lukasa's excellent and more elaborate answer for better and more details.

edited Jan 18 '17 at 08:27

answered Aug 15 '12 at 17:40

Daniel Stenberg

54,736
17
146
222

Thanks so much, @daniel-stenberg. So do you have a clue why the string I send is `"name" : "Pedro", "age" : "22"` and the string received is `{ '"name" : "Pedro", "age" : "22"' }`? – Pedro Baptista Afonso Aug 15 '12 at 17:49
1

If you didn't send the '{' and '}', then you didn't receive them. Then I assume your JSON handler that receives data adds them by itself for you as a service or similar. – Daniel Stenberg Aug 15 '12 at 18:25
@PedroBaptistaAfonso i think it is because of the header, the content type you are specifying. – code-jaff Aug 15 '12 at 18:25
@DanielStenberg, can you see the new example I added to the question? I'm not using any JSON handler, just sending a string. thank you so much. – Pedro Baptista Afonso Aug 15 '12 at 18:30
If I were you, I would packet capture the string sent by libcurl. I've just tested your code, and my server side got the correct data, as expected. – GergelyPolonkai Aug 15 '12 at 20:54
@W00d5t0ck I agree with you, that's a good sugestion... That's the best option to see if the problem is in the client or server side. – Pedro Baptista Afonso Aug 16 '12 at 08:56
@DanielStenberg this answer is incorrect. Without setting the correct header information, libcurl does NOT send the exact bytes. it wraps the bytes into a string that starts with {' , and ends with ':''}. – blorkfish Jan 18 '17 at 06:17
@DanielStenberg No in theory, maybe, but have you actually tried to compile and run this code sample ? If you did, then you will see the behaviour that is being described in the question. Maybe try it yourself before you disregard every other answer or comment. – blorkfish Jan 18 '17 at 07:02
4

I frigging wrote libcurl. I know what it does and doesn't. There is literally **nothing** in libcurl that adds braces to POST data. Nothing. You can claim that all you want. If such braces end up in the receiving end, they were added by something else than libcurl. – Daniel Stenberg Jan 18 '17 at 07:05
1

@blorkfish Daniel is right, and you are incorrect. See [my answer](http://stackoverflow.com/a/41714448/1401686) for a full explanation. – Lukasa Jan 18 '17 at 08:25
@DanielStenberg Hah, this is so funny. Being told off by the guy who actually wrote libcurl. Don't I look like a right idiot. Technically, the answer to the question "does libcurl append anything to the JSON string", the answer is a definite no, as I have learned. Yes it is the server that is modifying the request. Lukasa's answer gives the full picture. Given these symptoms, set the headers correctly is the simple answer. – blorkfish Jan 18 '17 at 12:44

score 1 · Answer 5 · answered Jul 07 '17 at 12:28

A key part of understanding whether the system is behaving correctly is seeing what the program is actually sending over the network. So another way to check the byte stream, instead of pointing it at your server (and/or running Wireshark), is to just run a netcat instance in a separate window on the test machine:

nc -l 8080

and point the code (CURLOPT_URL) at "http://localhost:8080".

You'll need to hit Control-D in the nc window to terminate the connection so that curl completes, but you can also type a test return text beforehand, which can be useful if you're expecting some sort of reply to test against.

score 0 · Answer 6 · answered Sep 11 '19 at 10:40

Had this same problem when posting data to node-red with json parser.
Solution for me was to treat string as a HTML

#include <curl/curl.h>

int main (int argc, char *argv[]) {
    CURL *curl;
    CURLcode res;

    curl_global_init(CURL_GLOBAL_ALL);
    curl = curl_easy_init();
    if (curl == NULL) {
        return 128;
    }
    struct curl_slist *headers = NULL;
    curl_slist_append(headers, "Content-Type: application/json");
    curl_easy_setopt(curl, CURLOPT_URL, #yourURL);
    curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "POST");
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "age=42&sex=male");

    res = curl_easy_perform(curl);

    curl_easy_cleanup(curl);
    curl_global_cleanup();
    return res;
}

Hope someone will find it helpful.

score -1 · Answer 7 · answered Oct 26 '16 at 08:57

-1

I used json-c for encoding and decoding jsons and it worked very well. The documentation is also easy to understand. https://linuxprograms.wordpress.com/2010/05/20/json-c-libjson-tutorial/

answered Oct 26 '16 at 08:57

Jonny_Pony

21
2

JSON requests in C using libcurl

7 Answers7

Linked

Related