Can't get python to compare lists properly

Question

I know it's impolite to include this much code in a question, but this is all necessary to explain the error. Apologies!

I'm writing a twitterbot, in python (twython), which is simply supposed to follow anybody who follows my account.

It reads in two text files, one of friends (people I'm following) and one of followers, using this code:

followers = []
friends = []
followers_old = []
friends_old = []

with open('followers_old.txt') as fo:
    followers_old=[L[:-1] for L in fo.readlines()]

with open('friends_old.txt') as fr:
    friends_old=[L[:-1] for L in fr.readlines()]

It then downloads the data from twitter, in chunks of 200. For each chunk, if the members are not already in the lists, it adds them, if any are already in the list, it doesn't append them, and it stops downloading chunks. This is done with the following code:

while(next_cursor):
        get_followers = twitter.get_followers_list(screen_name=username,count=200,cursor=next_cursor)
        time.sleep(60)
        for follower in get_followers["users"]:
                if follower not in followers_old:
                    followers.append(follower["screen_name"].encode("utf-8"))
                    next_cursor = get_followers["next_cursor"]
                else:
                    break

The above downloads followers. An identical bit of code does the same for friends.

Then, it should find members of 'followers' who are not members of 'friends', and follow them, using the following bit of code:

for fol in followers:
        if fol not in friends:
                twitter.create_friendship(fol)

Here is where I get an error. Twitter responds saying that I am trying to follow somebody I'm already following. I don't see how this can happen, given the 'if fol not in friends' line.

For those interested:

It finishes by appending the new followers and friends to the original text files read in at the start:

(For followers):

for fol in followers:
    fo.write("%s\n" % fol)

And then it does the same with friends.

Sorry for such a long question, I'd really appreciate the help.

Thanks, Alex.

EDIT:

Since there seems to be a bit of confusion caused by how I've summarised the question, here is the complete code:

twitter = Twython(CONSUMER_KEY,CONSUMER_SECRET,ACCESS_KEY,ACCESS_SECRET)

followers = [] 
friends = []
followers_old = []
friends_old = []

with open('followers_old.txt') as fo:
    followers_old=[L[:-1] for L in fo.readlines()]

with open('friends_old.txt') as fr:
    friends_old=[L[:-1] for L in fr.readlines()]

username = 'XXX'

next_cursor = -1
next_cursor_1 = -1

while(next_cursor):
        get_followers = twitter.get_followers_list(screen_name=username,count=200,cursor=next_cursor)
        time.sleep(60)
        for follower in get_followers["users"]:
                if follower not in followers_old:
                    followers.append(follower["screen_name"].encode("utf-8"))
                    next_cursor = get_followers["next_cursor"]
                else:
                    break

while(next_cursor_1):
        get_friends = twitter.get_friends_list(screen_name=username,count=200,cursor=next_cursor)
        time.sleep(60)
        for friend in get_friends["users"]:
                if friend not in friends_old:
                    friends.append(friend["screen_name"].encode("utf-8"))
                    next_cursor_1 = get_friends["next_cursor_1"]
                else:
                    break

for fol in followers:
        if fol not in friends:
                twitter.create_friendship(fol)

for fol in followers:
    fo.write("%s\n" % fol)

for fri in friends:
    fr.write("%s\n" % fri)

Based on the answers so far, I think the encoding is probably an issue. The files are initially blank, the script populates them entirely on its own, and updates them each time it runs.

I hope this makes it clearer, apologies for the ambiguity.

Answer 1

There are at least two problems in your code, and I don't know which one is causing your error (if not both). I'll explain one here; see this answer for the other.

if follower not in followers_old:
    followers.append(follower["screen_name"].encode("utf-8"))

It looks like some of your lists are of UTF-8 byte strings, while others are in Unicode strings. In Python 2.x, you can sometimes get away with mixing and matching, as long as you stick to strings that happen to be the same in your default charset and UTF-8 (generally, that means pure ASCII), but it will break as soon as you violate that.

For example:

>>> friends = [u'åbcdé']
>>> follower = 'åbcdé'
>>> follower in friends
False

So, you will try to follow people who are already your friends because you will think nobody is already a friend, unless their name is pure ASCII.

Answer 2

If you want to find the elements in followers that are not in friends use sets:

followers = ["foo","bar","foobar"]
friends = ["foo","foo1","bar1"]
print(set(followers).difference(friends))

set(['foobar', 'bar'])

Answer 3

There's another problem in your code, in addition to the Unicode problem.

• followers_old is the followers from the previous run.
• followers is only the new followers that weren't there last time.
• friends_old is the friends from the previous run.
• friends is only the new friends that weren't there last time.

So, when you do this:

for fol in followers:
    if fol not in friends:
        twitter.create_friendship(fol)

You're only skipping new followers that aren't new friends. But you need to also skip new followers that aren't old friends. In other words, you need:

    if fol not in friends and fol not in friends_old:

Or, better, you need a single list that combines the two. Even better if it's a set rather than a list, and even better if you just iterate over the set difference instead of checking each one by one:

for fol in set(followers) - set(friends) - set(friends_old):
    twitter.create_friendship(fol)