Polling an API endpoint - how do I retry when no JSON is returned?

Question

I'm polling an API endpoint using a while loop that checks for whether or not a .get() method on the JSON returns None:

    while requests.get(render_execution_url, headers=headers).json().get('finalized_at') is None:
        status = requests.get(render_execution_url, headers=headers).json().get('status')
        status_detail = requests.get(render_execution_url, headers=headers).json().get('status_detail')
        logger.info("status for {} is {}.  detailed status is {}".format(render_execution_url, status, status_detail))

The idea here is that we keep polling the endpoint until the "finalized_at" value is populated.

Unfortunately, we periodically get failures when the JSON doesn't exist at all:

 File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
   return _default_decoder.decode(s)
 File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
   obj, end = self.raw_decode(s, idx=_w(s, 0).end())
 File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
   raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

I've tried using the retry decorator on the method (see below for the decorator syntax) but it doesn't seem to be executing retries when I hit this failure.

@retry(stop_max_attempt_number=7, wait_fixed=10000)

Is there a graceful, Pythonic way to deal with the case when the JSON doesn't exist (i.e., to try again in some amount of time)?

Where would I put the try statement here? If I wrap the whole while statement in a try/except block, I won't get the retry behavior I want, unless I wrapped an additional while True: on top of it. — snark17, Mar 06 '19 at 16:57
Why do you keep making the same request over and over gain? Surely you want `status` and `status_detail` from the same response that didn't have `finalized_at` as a field? — chepner, Mar 06 '19 at 17:12

score 2 · Accepted Answer · edited Mar 06 '19 at 17:12

Your code is too dense to easily separate out the different conditions you need to handle, and so your error report doesn't make it clear exactly what "when the JSON doesn't exist at all" means - is the server returning 404 (Page Not Found), or is the response data empty, or something else?

Here's a re-write that doesn't access the URL for each access to the JSON. It may not suit your needs perfectly, but it should give you start.

while True:
    resp = requests.get(render_execution_url, headers=headers)
    # I assume response status is always 200 or 204 -
    # Really easy to detect a 404 here if that happens.
    if not resp.data:
        time.sleep(WAIT_TIME)
        continue
    rj = resp.json()
    if rj.get('finalized_at') is not None:
        break
    status = rj.get('status')
    status_detail = rj.get('status_detail')
    logger.info("status for {} is {}.  detailed status is {}"
                .format(render_execution_url, status, status_detail))
    time.sleep(WAIT_TIME)

This is very instructive, and exactly what I needed. Thank you! — snark17, Mar 06 '19 at 19:53

Polling an API endpoint - how do I retry when no JSON is returned?

1 Answers1