-2

I have encoutered a problem using python to retrieved followers information thanks to the twitter API. As you know API cut after a certain time out or rate limit reach. My idea was to cut the list of followers i have to crawl in pack of, for exemple 200 screen names, wait, and then go on. For that i use islice: 

while True:
lines =[x.rstrip('\n') for x in islice(followers, 200)]
for i in lines:
    try:
    # Request general user information
        resp = twitter.show_user(screen_name=i)
        print('Retrieving information for'+ ' '+str(i))
        spinner1.start()# Append fields to list
        user_info.append([resp['id'],
                resp['screen_name'],
                resp['name'],
                resp['lang'],
                resp['location'],
                resp['created_at'],
                resp['description'],
                resp['followers_count'],
                resp['friends_count'],
                resp['statuses_count'],
                resp['favourites_count'],
                resp['listed_count']])
        spinner1.stop()
        time.sleep(1)
    except:
        print('>>>>>' + 'This User: ' + str(x) + ' Is not accessible' + '<<<<<')
    time.sleep(6301)
if not lines:
    break

Problem is that the loop start again from the beginning of the list. I didn't succeed in making python understand to start from the point he has stopped. Any ideas? Thanks a lot!!!

Tristan Salord
  • 57
  • 1
  • 10
  • 1
    `except: print('>>>>>' + 'This User: ' + str(x) + ' Is not accessible' + '<<<<<')`: if _anything_ happens, you get this lame error message. Catch the proper exception and print the error from the exception instead – Jean-François Fabre Dec 13 '18 at 08:21
  • 1
    you should avoid using `try: ... except: ...` without specifying exception type since it will lead to undesired behavior like skipping `BaseException` subclasses (e.g. `KeyboardInterrupt`, which is probably not you meant to do) – Azat Ibrakov Dec 13 '18 at 08:23
  • @Jean-FrançoisFabre Thank you for this advice... i have effectively to look after how to catch the proper exception ^^I am new in python since... few time... I have received no specific education on it and try to learn observing how people do, and by my own mistakes. Thanks i have to correct this – Tristan Salord Dec 13 '18 at 08:37
  • @AzatIbrakov Okay Azatlbrakov, i'm very sorry but don't understand a word of your advice. I have to investigate python documentation and google to understand ^^ sorry – Tristan Salord Dec 13 '18 at 08:38

2 Answers2

0

No need for the islice() function, you can directly slice the list of followers and using a simple counter to keep track of where you are in the list;

cnt = 0
while True:
  lines =[x.rstrip('\n') for x in followers[cnt:cnt+200])]
  for i in lines:
    try:
    # Request general user information
        resp = twitter.show_user(screen_name=i)
        print('Retrieving information for'+ ' '+str(i))
        spinner1.start()# Append fields to list
        user_info.append([resp['id'],
                resp['screen_name'],
                resp['name'],
                resp['lang'],
                resp['location'],
                resp['created_at'],
                resp['description'],
                resp['followers_count'],
                resp['friends_count'],
                resp['statuses_count'],
                resp['favourites_count'],
                resp['listed_count']])
        spinner1.stop()
        time.sleep(1)
    except:
        print('>>>>>' + 'This User: ' + str(x) + ' Is not accessible' + '<<<<<')
    time.sleep(6301)
  cnt += 200
if not lines:
    break

After each loop you just add 200 to the counter and it will slice the list from where you last finished slicing.

Nordle
  • 2,915
  • 3
  • 16
  • 34
-2

You can use the yield keyword to process your list:

def process_followers():
    for i in range(0, len(followers), 200):
        lines = followers[i:i + 200]
        for i in lines:
            try:
                # Request general user information
                resp = twitter.show_user(screen_name=i)
                print('Retrieving information for' + ' ' + str(i))
                spinner1.start()  # Append fields to list
                user_info.append([resp['id'],
                                  resp['screen_name'],
                                  resp['name'],
                                  resp['lang'],
                                  resp['location'],
                                  resp['created_at'],
                                  resp['description'],
                                  resp['followers_count'],
                                  resp['friends_count'],
                                  resp['statuses_count'],
                                  resp['favourites_count'],
                                  resp['listed_count']])
                spinner1.stop()
                time.sleep(1)
                except:
                    print('>>>>>' + 'This User: ' + str(x) + ' Is not accessible' + '<<<<<')
        yield

Then you can use it as follow:

processor = process_followers()
try:
    while True:
        processor.__next__()
        time.sleep(6301)
except StopIteration:
    pass
T.Lucas
  • 848
  • 4
  • 8