Using urllib2 to make a post request with headers

Question

I want to send a post request to a URL with specific data and header types. using these two links I found that how to do that, but it's not working:
https://stackoverflow.com/questions/5693931/python-post-request
How do I send a custom header with urllib2 in a HTTP Request?
This is my code:

import urllib
import urllib2

url = 'https://clients6.google.com/rpc'
values = [
    {"method": "pos.plusones.get",
     "id": "p",
     "params": {
                "nolog": True,
                "id": "http://www.newswhip.com",
                "source": "widget",
                "userId": "@viewer",
                "groupId": "@self"
                },
     "jsonrpc": "2.0",
     "key": "p",
     "apiVersion": "v1"
    }]
headers = {"Content-type" : "application/json:"}

data = urllib.urlencode(values)
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read()
print the_page

I can get result with these values in Postman Rest Client. but after execution of this code the result is:

Traceback (most recent call last):
  File "D:/Developer Center/Republishan/republishan2/republishan2/test2.py", line 22, in <module>
    data = urllib.urlencode(values)
  File "C:\Python27\lib\urllib.py", line 1312, in urlencode
    raise TypeError
TypeError: not a valid non-string sequence or mapping object

I've also tried to use dictionary instead of list like this:

values = {"method": "pos.plusones.get",
     "id": "p",
     "params": {
                "nolog": True,
                "id": "http://www.newswhip.com",
                "source": "widget",
                "userId": "@viewer",
                "groupId": "@self"
                },
     "jsonrpc": "2.0",
     "key": "p",
     "apiVersion": "v1"
    }

It executes the script but the result contains error:

{"error":{"code":-32700,"message":"Unable to parse json","data":[{"domain":"global","reason":"parseError","message":"Unable to parse json"}]}}

As I told, I can execute script with list instead of dictionary with Postman Rest Client. Look at the results in Postman: enter image description here What should I do?

`values` should be dict not list. Try `values = {"method": "pos.plusones.get", ...., "apiVersion": "v1"}` — sk11, Jul 22 '14 at 08:22
Some server error, I assume. Check that the values you are sending are correctly formatted and the server expects it in the same fashion. — sk11, Jul 22 '14 at 12:49
Man these values are working with a Rest Client! I think an additional configuration in urllib needed. — ehsan shirzadi, Jul 22 '14 at 13:02
Just a wild guess, why u have the extra colon after `application/json` in `headers = {"Content-type" : "application/json:"}`? — sk11, Jul 22 '14 at 13:42
@sk11 I removed the colon, nothing happened, is it possible to copy and run my code? I know it's an over expectation but I really need that solution from an expert. — ehsan shirzadi, Jul 22 '14 at 13:58

score 4 · Accepted Answer · answered Jul 22 '14 at 14:11

It looks like urllib.urlencode doesn't understand nested dicts:

     In [38]: urllib.urlencode({"a": "asas", "df": {"sds": 123, "t": "fgfg"}})
     Out[38]: 'a=asas&df=%7B%27t%27%3A+%27fgfg%27%2C+%27sds%27%3A+123%7D'

Or your example:

     In [41]: urllib.urlencode(values)
     Out[41]: 'jsonrpc=2.0&apiVersion=v1&id=p&params=%7B%27nolog%27%3A+True%2C+%27source%27%3A+%27widget%27%2C+%27userId%27%3A+%27%40viewer%27%2C+%27id%27%3A+%27http%3A%2F%2Fwww.newswhip.com%27%2C+%27groupId%27%3A+%27%40self%27%7D&key=p&method=pos.plusones.get'

See, braces in "params" gets messed up.

I'm not sure how to cure this using urllib. So i'll recommend requests library. http://docs.python-requests.org/en/latest/user/quickstart/#custom-headers

in short it will look like this (you'll need to install requests library first, for example using pip: pip install requests):

     import requests
     import json

     url = 'https://clients6.google.com/rpc'
     values = {
         "method": "pos.plusones.get",
         "id": "p",
         "params": {
                    "nolog": True,
                    "id": "http://www.newswhip.com",
                    "source": "widget",
                    "userId": "@viewer",
                    "groupId": "@self"
                   },
          "jsonrpc": "2.0",
          "key": "p",
          "apiVersion": "v1"
     }
     headers = {"content-type" : "application/json"}

     req = requests.post(url, data=json.dumps(values), headers=headers)
     print req.text

It works for me.

It was all about this snippet that made the difference for me "data=json.dumps(values)" as I even did not have a dict JSON but a simple object and it was failing to send it properly using requests. Thanks! — DigitalFox, May 07 '20 at 14:35

Using urllib2 to make a post request with headers

1 Answers1