Python error in parsing JSON

Question

i am developing a machine learning algorithm to predict temperature in my city. I am using wunderground API in order to obtain the data. So, in this piece of code where I execute a request to the API:

def extract_weather_data(url, api_key, target_date, days):  
records = []
for _ in range(days):
    request = BASE_URL.format(API_KEY, target_date.strftime('%Y%m%d'))
    response = requests.get(request)
    print request
    if response.status_code == 200:
        data = response.json()['history']['dailysummary'][0]
        records.append(DailySummary(
            date=target_date,
            meantempm=data['meantempm'],
            meandewptm=data['meandewptm'],
            meanpressurem=data['meanpressurem'],
            maxhumidity=data['maxhumidity'],
            minhumidity=data['minhumidity'],
            maxtempm=data['maxtempm'],
            mintempm=data['mintempm'],
            maxdewptm=data['maxdewptm'],
            mindewptm=data['mindewptm'],
            maxpressurem=data['maxpressurem'],
            minpressurem=data['minpressurem'],
            precipm=data['precipm']))
    time.sleep(6)
    target_date += timedelta(days=1)
return records  

records = extract_weather_data(BASE_URL, API_KEY, target_date, 100)

I obtain this error after 3-4 request:

 Traceback (most recent call last):
 File "data.py", line 45, in <module>
 records = extract_weather_data(BASE_URL, API_KEY, target_date, 100)
 File "data.py", line 26, in extract_weather_data
 data = response.json()['history']['dailysummary'][0]
 File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 892, in json
 return complexjson.loads(self.text, **kwargs)
 File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
 return _default_decoder.decode(s)
 File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
 File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode
 obj, end = self.scan_once(s, idx)
 ValueError: Expecting property name enclosed in double quotes: line 11 column 1 (char 155)

How can i fix this?

Thanks you a lot!

It looks like the JSON might be malformed. Can you give a portion of the response without trying to convert it? — roganjosh, Jun 16 '18 at 09:37
Perfect, thanks. And now I'm totally confused because that link has just given me two completely different responses. One in HTML and one in JSON format simply by clicking it twice. — roganjosh, Jun 16 '18 at 09:39
I understand that, but the server itself has just given me a definitely-not-json response the first time I clicked the link. Now it gives JSON. There's something not right with the API. — roganjosh, Jun 16 '18 at 09:41
I have see now this problem, how can you suggest to fix it? Maybe i can re-try the request if it doesn't work — user3836982, Jun 16 '18 at 09:45
Honestly I don't know. Probably it should be raised as an issue with them. In terms of your own code, you probably just have to loop in a `try`/`except`. The API should not behave that way. — roganjosh, Jun 16 '18 at 09:47

score 3 · Accepted Answer · answered Jun 16 '18 at 10:03

Golden rule when it comes to HTTP requests (API or not and whatever the expected response's content type) is that things can go wrong in many ways and at any time and that they WILL go wrong one day or another in the most unexpected way, so if you hope to have a decenty robust client program, you have to be prepared to handle just any possible outcome.

Most often, this actually means wrapping requests calls in try/except handlers plus testing the response's status code, content-type and effective content (instead of blindly assuming you got what you asked for), and depending on the exception / unexpected response value decide whether it's worth retrying the request a couple times (with increasing delays between the retries) before giving up (some error conditions can be temporary) or just give up immediatly. In all cases, you also want to make sure you let the user know what went wrong, with as much informations as possible (request, response if you got one, exception and full traceback else) when you decide to give up.

I think this is the only reasonable answer. I don't seem to be able to recreate the behaviour I observed so unfortunately I can't give the OP a test case to pass on to the API creators. It looks totally confined within something going on at their end :/ — roganjosh, Jun 16 '18 at 10:08

DrRelling · Answer 2 · 2018-06-16T10:10:42.730

0

From the last part of your error:

ValueError: Expecting property name enclosed in double quotes: line 11 column 1 (char 155)

it sounds like the data you're receiving isn't in the right format - either it's not JSON, or it's not quite the way Python expects. If you just print response.content or get the data through something like Postman, that might help you to work out where the problem is.

Depending on what you're receiving, you may be able to "fix" the response before parsing it as JSON, or you may need to parse it a different way.

edited Jun 16 '18 at 10:10

answered Jun 16 '18 at 09:46

DrRelling

122
1
4
10

The API looks broken to me. It will either give HTML or JSON with the same request. – roganjosh Jun 16 '18 at 09:47
@roganjosh I've tried getting the data with a basic requests.get a few times now, and it returns JSON every time for me. Given it's an intermittent thing it could just be I'm getting lucky and not running into it, but what HTML did you get? – DrRelling Jun 16 '18 at 09:59
Well now I can't recreate it so I've asked in chat if others can. But for sure it was some form of HTML with a bunch of `
` tags. I'm trying to believe I've not gone insane and imagined the response I got :) The only reason I happened to click it twice was because I went back to the question to tell the OP it was clearly not JSON, then decided to re-check.

roganjosh

Jun 16 '18 at 10:01

Python error in parsing JSON

2 Answers2