Catching exceptions raised in the greenlets

Question

I'm trying to catch the exceptions raised within the greenlets. According to this tutorial, unfortunately 'exceptions raised in the Greenlet, stay inside the Greenlet'. In the code below, I have a sync method which spawns crawl greenlets. Those crawl greenlets raise HTTPError exception, which I need to catch in the body of sync method. Based on the exception, in the sync method I will set the appropriate status of the Account object. Any idea how can I catch exceptions raised in greenlets in the sync method in order to accomplish that? Thank you in advance!

# this is standalone helper method
def crawl(item):
    try:
        item.refresh_children(False) # THROWS HTTPError exception
        greenlets = [gevent.spawn_link_exception(crawl, child) for child in item.children]
        gevent.joinall(greenlets)
    except HTTPError, e:
        print e.message
        raise e
    except (JSONDecodeError, InvalidCredentialsException) as e:
        print e.message
        raise e

# this is instance method of the Account class 
def sync(self):
    "Sync search index with data from source"
    try:
        greenlets = [gevent.spawn_link_exception(crawl, item) for item in self.get_navigation()]
        gevent.joinall(greenlets)
        self.date_synced = datetime.datetime.now()
        self.save()
    except HTTPError, e:
        if e.status_code == 401:
            self.status = 'revoked'
        else:
            self.status = 'error'
        self.save()
    except LinkedFailed, e:
        print e.message
        exception_name = e.message.split()[-1]
        if exception_name in ['HTTPError', 'JSONDecodeError']:
            self.status = 'error'
            self.save()
        elif exception_name == 'InvalidCredentialsException':
            self.status = 'revoked'
            self.save()

score 3 · Answer 1 · answered Apr 21 '13 at 15:40

Greenlets will hold the results of the function spawned for the greenlet or the exception, if one was raised. You can get the result, or re-raise the exception, with the get method. Since you seem to be changing the status based on any of the greenlets erroring the sync method would wind up looking something like:

def sync(self):
    "Sync search index with data from source"
    greenlets = [gevent.spawn_link_exception(crawl, item) for item in self.get_navigation()]
    gevent.joinall(greenlets)
    try:
        results = [greenlet.get() for greenlet in greenlets]
    except HTTPError, e:
        if e.status_code == 401:
            self.status = 'revoked'
        else:
            self.status = 'error'
        self.save()
    except LinkedFailed, e:
        print e.message
        exception_name = e.message.split()[-1]
        if exception_name in ['HTTPError', 'JSONDecodeError']:
            self.status = 'error'
            self.save()
        elif exception_name == 'InvalidCredentialsException':
            self.status = 'revoked'
            self.save()

    self.date_synced = datetime.datetime.now()
    self.save()

Catching exceptions raised in the greenlets

1 Answers1