extract data from a dictionary returned by pycurl

Question

I have this:

import pycurl
import pprint
import json

c = pycurl.Curl()
c.setopt(c.URL, 'https://mydomainname.com')

c.perform()

the above code return a dictionary like this:

{"name":"steve", "lastvisit":"10-02-2012", "age":12}

I want to loop through that dictionary and get just the age:

age : 12

I tried:

diction = {}
diction = c.perform()
pprint.pprint(diction["age"])

No data returned and I got this error:

TypeError: 'NoneType' object is unsubscriptable

I am presuming you are using Python 3 (judging by the exception, python 3.1 or 3.2). — Martijn Pieters, Mar 16 '13 at 19:35

Martijn Pieters · Accepted Answer · 2020-03-04T21:38:35.333

18

c.perform() doesn't return anything, you need to configure a file-like object to capture the value. A BytesIO object would do, you can then call .getvalue() on that after the call completes:

import pycurl
import pprint
import json
from io import BytesIO

c = pycurl.Curl()
data = BytesIO()

c.setopt(c.URL, 'https://mydomainname.com')
c.setopt(c.WRITEFUNCTION, data.write)
c.perform()

dictionary = json.loads(data.getvalue())
pprint.pprint(dictionary["age"])

If you are not married to pycurl, you might find requests to be a lot easier:

import pprint
import requests

dictionary = requests.get('https://mydomainname.com').json()
pprint.pprint(dictionary["age"])

Even the standard library urllib.request module will be easier than using pycurl:

from urllib.request import urlopen
import pprint
import json

response = urlopen('https://mydomainname.com')
dictionary = json.load(response)
pprint.pprint(dictionary["age"])

edited Mar 04 '20 at 21:38

answered Mar 16 '13 at 19:23

Martijn Pieters

1,048,767
296
4,058
3,343

Thank you very much Metijn, I tried the code and I got this error: `AttributeError: getvalue` – mongotop Mar 16 '13 at 19:26
1

@mongotop: Apologies, I didn't read the [documentation](http://pycurl.sourceforge.net/doc/curlobject.html) in enough detail. Why are you using `pycurl`? Why not use a more convenient library? – Martijn Pieters Mar 16 '13 at 19:27
To be honest that is the one I know about, can you please suggest another library. – mongotop Mar 16 '13 at 19:29
Your code works like a charm!!!! I'm reading now about requests. Thank you very much for educating me!!!! highly appreciated!!! – mongotop Mar 16 '13 at 19:40
@MartijnPieters is it possible to use `requests` to imitate `curl 'some_url' -d 'some_json'`? I couldn't find a way to do that. – Maroun May 14 '15 at 14:39
@MarounMaroun: you mean `requests.post('some_url', json=python_object)`? It'll set the right headers and encode the Python structure to JSON for you. – Martijn Pieters May 14 '15 at 14:54
@MartijnPieters Thanks. I'm actually trying to query an *elastic search* server, by `data=my_json` it's giving an error, I think it's because `'`s instead of `"`s from `json.loads`. I'll be grateful if you can enlighten me on this. – Maroun May 14 '15 at 15:07
@MarounMaroun: that'd be a new question or the chat room, not the comments here. – Martijn Pieters May 14 '15 at 15:08
@MartijnPieters 'use requests instead' isn't an answer to this problem. it performs extremely badly with large responses, so for me, the only option is to use pycurl. – Strobe_ Apr 12 '17 at 10:55
@Strobe_: Do read the other comments, where the OP is asking for recommendations. As for large responses: have you tried streaming the response? – Martijn Pieters Apr 12 '17 at 10:57

extract data from a dictionary returned by pycurl

1 Answers1