Python requests module sends JSON string instead of x-www-form-urlencoded param string

Question

I was under the impression that POSTSs using x-www-form-urlencoded specifications should send a URL encoded param string in the body of the post. However, when I do this

data = json.dumps({'param1': 'value1', 'param2': 'value2'})
Requests.post(url, data=data)

The body of the request on the receiving end looks like this:

{"param1": "value1", "param2": "value2"}

But I was expecting to get this

param1=value1&param2=value2

How I can get Requests to send the data in the second form?

score 175 · Accepted Answer · edited Mar 13 '20 at 11:11

The reason you're getting JSON is because you're explicitly calling json.dumps to generate a JSON string. Just don't do that, and you won't get a JSON string. In other words, change your first line to this:

data = {'param1': 'value1', 'param2': 'value2'}

As the docs explain, if you pass a dict as the data value, it will be form-encoded, while if you pass a string, it will be sent as-is.

For example, in one terminal window:

$ nc -kl 8765

In another:

$ python3
>>> import requests
>>> d = {'spam': 20, 'eggs': 3}
>>> requests.post("http://localhost:8765", data=d)
^C
>>> import json
>>> j = json.dumps(d)
>>> requests.post("http://localhost:8765", data=j)
^C

In the first terminal, you'll see that the first request body is this (and Content-Type application/x-www-form-urlencoded):

spam=20&eggs=3

… while the second is this (and has no Content-Type):

{"spam": 20, "eggs": 3}

For me this answer causes this error: https://stackoverflow.com/questions/45037676/python-3-6-urllib-typeerror-cant-concat-bytes-to-str — Alex R, Feb 27 '19 at 03:56
json payload can also be sent as requests.post(url, json=d), my two and a half cents. — a_secenthusiast, Feb 08 '23 at 01:29

score 14 · Answer 2 · edited Nov 20 '21 at 20:37

14

Important to add, it does not work for nested json So, if you have

# Wrong
data = {'param1': {'a':[100, 200]},
        'param2': 'value2',
        'param3': False}

# You have to convert values into string:
data = {'param1': json.dumps({'a':[100, 200]}),
        'param2': 'value2',
        'param3': json.dumps(False)}

edited Nov 20 '21 at 20:37

DeyaEldeen

10,847
10
42
75

answered Sep 18 '19 at 19:28

NotTooTechy

448
5
9

When you say `it` does not work for nested json, what are you referring to? I have a dict with a nested dict and I am passing it - the `dict` so it should be form data, right? Does that not work for the nested dict? How about a list inside of the nested dict? I wouldn't want to do what you were doing since that is turning everything into json and it seems the other answers says to leave it as a dict; specifically don't change it into json. – Cfomodz Jul 20 '21 at 23:20
You have to send like {"param1": "value1", "param2": "value2"}, json.dumps converts a dictionary into string. That's all. – NotTooTechy Jul 26 '21 at 20:38
so I am trying to send a dict of of a 'saved search' for real estate - the persons email who wants the email update, their id, etc., but then also in that is a dict of criteria - how many beds, baths, etc. and there are like 40-50 criteria they can choose from. The only way I have found to do this is to append `criteria['min_beds']:3, criteria['min_baths']:2` etc. etc. etc. Is there no way to turn the `criteria` `dict` into its parts? – Cfomodz Jul 27 '21 at 18:33
The subject is sending POSTSs using x-www-form-urlencoded server. Typical server that expects x-www-form-urlencoded form receives a dictionary as described {'strParam1':'strVal1', 'strParam2':'strVal2'} that's all. – NotTooTechy Jul 28 '21 at 15:51

Mercury · Answer 3 · 2022-09-11T12:51:18.097

Short answer with example:

import requests

the_data = {"aaa": 1, "bbb": 2, "ccc": "yeah"}
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
# Execute the post
requests.post("http://bla.bla.example.com", data=the_data, headers=headers)
# You have POSTed this HTTP body: aaa=1&bbb=2&ccc=yeah (note, although the content-type is called urlencoded the data is not in the URL but in the http body)
# to this url: "http://bla.bla.example.com"

Requests library does all the JSON to urlencoded string conversion for you

References:

MDN Web docs, Requests lib post url-encoded form

Python requests module sends JSON string instead of x-www-form-urlencoded param string

3 Answers3

Linked