i am trying crawl all tweets of a hashtag #nationaldoughnutday but failing to do so due to rate limit.
with reference to the code below, i tried to put the code in a while loop so that when rate limit resets, i can resume scraping from the last crawled date (until_date)
However i kept getting this error repeatedly and my crawler does not seem to restart crawling after sleeping for a long time.
TweepError Failed to send request: ('Connection aborted.', error (10054, 'An existing connection was forcibly closed by the remote host'))
Sleeping...
TweepError Failed to send request: ('Connection aborted.', error (10054, 'An existing connection was forcibly closed by the remote host'))
Sleeping...
TweepError Failed to send request: ('Connection aborted.', error (10054, 'An existing connection was forcibly closed by the remote host'))
I have tried to removing the inner try catch loop but didn't help too
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth, wait_on_rate_limit=True,wait_on_rate_limit_notify=True)
query = '#nationaldoughnutday'
untill_date = '01-07-2019'
while True:
try: #outer try catch
tweets = tweepy.Cursor(api.search, q=query + '-filter:retweets', rpp=100, lang='en',tweet_mode='extended',until = until_date).items()
for tweet in tweets:
try: #inner try catch
print "tweet : ", tweet.created_at
#this is so that if i reconnect with cursor, i will start with the date before the last crawled tweet
until_date = tweet.created_at.date() - datetime.timedelta(days=1)
except tweepy.TweepError as e:
print 'Inner TweepyError', e
time.sleep(17 * 60)
break
except tweepy.TweepError as e:
print 'Inner TweepyError',
print "sleeping ...."
time.sleep(17 * 60)
continue
except StopIteration:
break
Thank you in advance!
