Making HTTP POST request using requests in Python

Question

I am trying to make a HTTP request to an API using the POST method. The API that I am using is meant to take in three parameters (key1, key2, key3) and return a json file. Unfortunately, my POST request does not seem to return anything when I am using the data method to pass my dictionary to the API. This seems to be very strange because it seems to work when I am using the params method. I cannot try to understand this as this process seems to be very opaque (e.g. I cannot a URL to see how the payload are passed onto the API).

My question: What am I doing wrong here?

POST request where the parameters are sent over to the API using data method:

import requests

url = 'http://example.com/API.php'
payload =  {
            'key1': '<<Contains very long json string>>', 
            'key2': 5, 
            'key3': 0
           }

print len(str(payload)) # Prints 6717
r = requests.post(url, data=payload) << Note data is used here
print r.status_code # Prints 200
print r.text # Prints empty string

POST request code where the parameters are sent over to the API using the params method:

import requests

url = 'http://example.com/API.php'
payload =  {
            'key1': '<<Contains very long json string>>', 
            'key2': 5, 
            'key3': 0
           }

print len(str(payload)) # Prints 6717
r = requests.post(url, params=payload) << Note here params is used here
print r.status_code # Prints 200
print r.text # Prints expected JSON results

If you are wondering why I would like to use the data method over params... I am trying to pass other dictionaries containing longer strings and the params method does not seem to do it because I am getting ERROR 414. I was hoping that the error could be resolved by using data.

The API that I am using was written in PHP.

What encoding does the API expect for the request body? Could it be expecting JSON, or multipart/form-data? — Martijn Pieters, Jun 25 '17 at 12:38
Possible duplicate of [Difference between "data" and "params" in Python requests?](https://stackoverflow.com/questions/24535920/difference-between-data-and-params-in-python-requests) — agtoever, Jun 25 '17 at 15:15
I am sorry. At the moment, I am not entirely sure about this as the API was written in PHP by my colleague. However, I was going under the assumption that the API was expecting a JSON. I have tried the adding the following headers to the HTTP request along with the **data** method: `headers = {'Content-Type': "application/json; charset=UTF-8"}` and `headers = {'Content-Type': "multipart/form-data; charset=UTF-8"}` They didn't seem to help. I hope I answered your question. — user2646237, Jun 25 '17 at 16:47

agtoever · Answer 1 · 2017-06-25T16:52:15.717

1

Short answer
This is because params sends the parameters as part of the http POST request, while data sends them as part of the body of the request. In your case: just call the api using params and you're fine. This is absolutely normal (and expected) behaviour.

Demonstration
Just start two commandlines. On the first, run netcat: nc -l 8888. On the other commandline, run python:

>>> import requests
>>> requests.post('http://localhost:8888',data={'a':1,'b':'2'})

At the netcat-side, we see the following request:

POST / HTTP/1.1
Host: localhost:8888
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.18.1
Content-Length: 7
Content-Type: application/x-www-form-urlencoded

a=1&b=2

Next, try the params way:

>>> requests.post('http://localhost:8888',params={'a':1,'b':'2'})

Netcat reports:

POST /?a=1&b=2 HTTP/1.1
Host: localhost:8888
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.18.1
Content-Length: 0

Note the differences in the first and last line.

As you can read from the documentation (italic emphasis is mine):

params -- (optional) Dictionary or bytes to be sent in the query string for the Request.
data -- (optional) Dictionary or list of tuples [(key, value)] (will be form-encoded), bytes, or file-like object to send in the body of the Request.

edited Jun 25 '17 at 16:52

answered Jun 25 '17 at 15:13

agtoever

1,669
16
22

Thank you for explaining the difference between **params** and **data**. I do not understand why sending my dictionary over to the API as part of the body does not return anything while sending it part of the HTTP request works. Do you think it could be a problem arising on the API side? – user2646237 Jun 25 '17 at 16:28
No. This is perfectly normal behaviour in such cases. The (http server of the) API just looks at the posted url and extracts the parameters from there. It probably ignores the content (data) of the post. Posts with data are common when uploading forms with files. The form is sent like with "params". The file is send using "data". – agtoever Jun 25 '17 at 16:49
Just use params when calling your API. I'll add this to my answer, because that answers that last bit of your question – agtoever Jun 25 '17 at 16:51
I am working with multiple dictionaries, with each one representing a document. I am sending each dictionary to the API to process some information about the document and return results. Using **params** works for some dictionaries and doesn't work for others. For example, for a document `print len(str(payload))` prints 18563 - this is clearly a **very large dictionary**. As you can expect, trying to make the HTTP request using *params* throws back the following error: 414 Request-URI Too Large - The requested URL's length exceeds the capacity limit for this server. – user2646237 Jun 25 '17 at 17:06
In that case, you should see what other option this API has. [RFC 2616](http://www.faqs.org/rfcs/rfc2616.html) states that there is no a-priori restriction on uri length, so it depends on the webserver you hare talking to. 8KB (8192 bytes) is a common limit. Anything beyond that size should be handled differently than putting all that document data inside the uri. Without more context, all I can suggest is to check the documentation of the API. – agtoever Jun 25 '17 at 18:14
The problems that I have been experiencing seems to be consistent with your last hypothesis. Anything below 8000 characters (or ~8KB) seems to be fine and anything beyond that, I seem to be experiencing error 414. Since the API is hosted on a webserver that I have control over, would it be possible to increase the 8KB? I am aware that this is not a bad practice. However, at this stage I am testing whether my script works as expected. – user2646237 Jun 25 '17 at 21:19
That question [already has an answer here](https://stackoverflow.com/questions/2586339/how-to-increase-apache-2-uri-length-limit) on Stack Overflow. – agtoever Jun 26 '17 at 04:36

Making HTTP POST request using requests in Python

1 Answers1