Python urllib.request.Request parameter 'data' object type

Question

I try to use urllib.request.Request (Python 3.6.7) to make an API call to a internal web services to get some json results. I need to send some data and headers to the server, so I use the urllib.request.Request class to do this. For the input of data, I try to find out what is the format it will accept. From the Python docs, it says:

The supported object types include bytes, file-like objects, and iterables.

So I use a dictionary data type for this parameter data. Here is my code:

import urllib

my_url = "https://httpbin.org/post"
my_headers = { "Content-Type" : "application/x-www-form-urlencoded" }
my_data = {
        "client_id" : "ppp",
        "client_secret" : "000",
        "grant_type" : "client_credentials" }

req = urllib.request.Request(url=my_url, data=my_data, headers=my_headers)
response = urllib.request.urlopen(req)
html = response.read()
print(html)

I then get error like this:

Traceback (most recent call last):
  File "./callapi.py", line 23, in <module>
    response = urllib.request.urlopen(req)
  File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib64/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/lib64/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1064, in _send_output
    + b'\r\n'
TypeError: can't concat str to bytes

I then follow the example in this docs page, and change my code to:

import urllib

my_url = "https://httpbin.org/post"
my_headers = { "Content-Type" : "application/x-www-form-urlencoded" }
my_data = {
        "client_id" : "ppp",
        "client_secret" : "000",
        "grant_type" : "client_credentials" }

my_uedata = urllib.parse.urlencode(my_data)
my_edata = my_uedata.encode('ascii')

req = urllib.request.Request(url=my_url, data=my_edata,headers=my_headers)
response = urllib.request.urlopen(req)
html = response.read()
print(html)

it then works.

My question is, isn't it in the docs it says this class accept data type iterables ? why does my parameter in dict is wrong ? My final result that is working use str.encode() method which returns an byte object, and it seems this class must take a byte object and not an iterables object.

I am trying to use Python Standard Library docs as the main source of reference to code in Python, however I am having a hard time to use it, hope anybody can shed some light in helping me to understand more on how the library docs works, or if there is any other tutorial I need to go through before I can use it in a better way. Thanks.

On my version of python (Python 3.5.2) I get the following error: `ValueError: Content-Length should be specified for iterable data of type {'client_id': 'ppp', 'grant_type': 'client_credentials', 'client_secret': '000'}` so the issue is not the type but that you must specify `Content-Length` when using an iterable. Which python version are oyu using? Maybe older versions simply had a worse error message (in the traceback I see the original exception is about an issue with bytes) — Giacomo Alzetta, Feb 21 '19 at 08:29
@GiacomoAlzetta: I have both 2.7 and 3.6 in my Centox box, but I use **alternatives** to make it switch to 3.6, so when I do `python -V` I got a 3.6.7. — sylye, Feb 21 '19 at 08:42
Now I check ver3.5 library [docs](https://docs.python.org/3.5/library/urllib.request.html#module-urllib.request) :`The urllib.parse.urlencode() function takes a mapping or sequence of 2-tuples and returns an ASCII string in this format. It should be encoded to bytes before being used as the data parameter.`, it seems the Python3.6.7 behaviour is more like 3.5 docs :( anything I missed? I now also tried put in **Transfer-Encoding: chunked** in headers following 3.6 docs but it still didn't work. What's wrong is it? — sylye, Feb 21 '19 at 09:15
this seems like a bug in the docs. They dropped the fact that the iterable must be an iterable of bytes and not strings. — Giacomo Alzetta, Feb 21 '19 at 09:37
I have just filed a bugs for this docs issue in Python bug tracker: https://bugs.python.org/issue36064 . Hope that helps. — sylye, Feb 21 '19 at 10:26
This problem seems already been discussed back in 2015 in another Python bug issue, the discussion still go on over there: https://bugs.python.org/issue25439 — sylye, Feb 22 '19 at 03:32

score 1 · Answer 1 · answered Feb 21 '19 at 09:42

I agree with you, the doc is not explicit. What is implicit is that if the data parameter is an iterable, it must be an iterable of bytes. When I have tried to pass a string as data I got an explicit error message:

TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

So for that reason, the iterable cannot be a dictionnary. As an exemple of valid iterable that in not a byte object (ok, just an example, no reason to use that in real code...):

def dict_iter(d):
    for i in d.items():
        yield(str(i).encode())

You can use that generator for the data parameter:

req = urllib.request.Request(url=my_url, data=dict_iter(my_data), headers=my_headers)

What you said is correct and it seems to not only related to Python docs only, there are now discussion over this [bug tracker](https://bugs.python.org/issue36064) and a previous [2015 bug](https://bugs.python.org/issue25439). So I will see what is the outcome from that two bug discussion. — sylye, Feb 22 '19 at 03:38

Python urllib.request.Request parameter 'data' object type

1 Answers1