How can I improve my error handling so the exception StopIteration in Tweepy is processed correctly and execution can continue?

Question

I have the following function to get the twitter followers and write them to a MySQL database. My problem is my error handling doesn't handle the StopIteration case very well. When I execute the code it does write the details to the database in line with the API restrictions but at the end it generates the error below so no further code is executed. How can I improve my error handling so the exception is handled correctly ?

StopIteration: The above exception was the direct cause of the following exception: RuntimeError

def limit_handled(cursor):
    while True:
        try:
            yield cursor.next()
        except tweepy.RateLimitError:
            time.sleep(15 * 60)

def writeFollowersToDB(TwitterAPI,DBConnection,SocialHandle ="Microsoft",DatabaseTable="twitter"):
    AboutMe = TwitterAPI.get_user(SocialHandle)
    #print(AboutMe)
    DBCursor=mydb.cursor()
    #Create the SQL INSERT
    SQLInsert="INSERT INTO "+ DatabaseTable + " (SourceHandle,SourceHandleFollowersCount,SourceHandleFollowingCount, Action,DestinationHandle,DestinationHandleFollowersCount,DestinationPublishedLocation,DestinationWeb,CrawlDate) VALUES (%s, %s, %s,%s,%s,%s,%s,%s,%s) ;"
    print(SQLInsert)
    for follows in limit_handled(tweepy.Cursor(TwitterAPI.followers,id=SocialHandle).items()):
        today = date.today()
        try:
            if not follows.url:
                expandedURL =""
            else:
                #print(follows.url)
                expandedURL = follows.entities["url"]["urls"][0]["expanded_url"]
            #print(follows.screen_name, AboutMe.followers_count,AboutMe.friends_count,"from ", follows.location,"with ", " followers "," and provided this expanded URL: ",expandedURL )
            CrawlDate = today.strftime("%Y-%m-%d")
            #Insert into table
            SQLValues =(AboutMe.screen_name,AboutMe.followers_count,AboutMe.friends_count,"isFollowedBy",follows.screen_name,follows.followers_count,follows.location,expandedURL,CrawlDate)
            DBCursor.execute(SQLInsert,SQLValues)
            DBConnection.commit()
            print(AboutMe.screen_name,follows.screen_name,follows.followers_count)
        except StopIteration:
            DBConnection.close()
            break
        except:
            print(e.reason)
            DBConnection.close()
            break



---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
<ipython-input-2-d095a0b00b72> in limit_handled(cursor)
      3         try:
----> 4             yield cursor.next()
      5         except tweepy.RateLimitError:

C:\Path\site-packages\tweepy\cursor.py in next(self)
    194             # Reached end of current page, get the next page...
--> 195             self.current_page = self.page_iterator.next()
    196             self.page_index = -1

C:\Path\site-packages\tweepy\cursor.py in next(self)
     69         if self.next_cursor == 0 or (self.limit and self.num_tweets == self.limit):
---> 70             raise StopIteration
     71         data, cursors = self.method(cursor=self.next_cursor,

What do you want to do when `StopIteration` exception is fired? do you want to stop the for loop? On a side note, in the second exception catching you need to write `except Exception as e: ` otherwise `e` does not exist in `print (e.reason)` which in any case must be replace by `print(str(e))` — Jean-Marc Volle, Jul 05 '20 at 10:12
I want the function to end gracefully so my code execution can continue once the function returns. I suppose its to commit any outstanding data, close the db connection, exit the for loop, exit the function. Is it just a case of missing a "return" statement? What I don't understand is if I have the StopIteration exception why when the exception is fired does my function still cause python to stop — mobcdi, Jul 05 '20 at 10:42
I went to http://docs.tweepy.org/en/v3.8.0/code_snippet.html and saw an implementation of `limit_handled`. Assuming that is what you are using, I don't understand why you would even get a `StopIteration` exception in your code to begin with (or at least one that is not automatically handled by the `for follows in` statement). Add an `import traceback` statement and in your `StopIteration` exception handler add `print(traceback.format_exc())` to output a stack trace. — Booboo, Jul 07 '20 at 14:23
Exactly I'm using the limit_handling from tweepy. I've added it and the stack trace. It would seem to be the limit_handling thats actually raising the error so would a solution be to add the StopIteration handing to the limit function? — mobcdi, Jul 09 '20 at 09:40

score 0 · Answer 1 · answered Jul 10 '20 at 08:42

The issue was the limit_handling function didn't try to catch the StopIteration Error it was throwing. Improved function below works for me. Thanks to @Booboo for their help

def limit_handled(cursor):
    while True:
        try:
            yield cursor.next()
        except StopIteration:
            return
        except tweepy.RateLimitError:
            time.sleep(15 * 60)

How can I improve my error handling so the exception StopIteration in Tweepy is processed correctly and execution can continue?

1 Answers1