Execute tasks in a particular order

Question

I have this loop:

listGames = []
for home in range(totalPlayers - 1):
    for away in range(home + 1, totalPlayers):
        listGames.append((home, away))

print listGames
match_num = 1
for game in listGames:
    player1 = listPlayers[game[0]]
    player2 = listPlayers[game[1]]
    do_stuff(player1, player2)

When there are a lot of players, this loop can take quite some time, so is want to use threads to complete the loop faster. However, player1 and player2 are instances of classes, so doing stuff with them simultaneously would be bad. EDIT: The order in which these 'tasks' are executed does not matter otherwise.

I found http://www.troyfawkes.com/learn-python-multithreading-queues-basics/ which seems to be exactly what I want, but I am unsure how to adapt it to make sure that only 1 instance of the class/player is being run at once

(Simple) example:

totalPlayers = 4

0, 1, 2, 3
listGames = [(0, 1), (0, 2), (0, 3), (1, 2), (1, 3), (2, 3)]

so, game (0, 1), (2, 3) can be executed simultaneously, but the others will have to wait until these are done

Hints/Ideas?

Answer 1

Unless do_stuff() is IO-bound, this will probably make your code slower because of the global interpreter lock. Based on your statement that "this can take time when there are many players," I'm inclined to think that your program is probably CPU bound --- in this case, multithreading will probably harm your performance.

Speaking to your original question, you're asking for an exact cover of your set of players from among two element subsets -- which, unfortunately, is NP-complete.

Answer 2

Here's a sample program that shows a way you can do this. The idea is to create a multiprocessing.Pool to run many instances of do_stuff simultaneously. We also maintain a set that keeps track of all the players currently being processed by a do_stuff instance, so that we never process the same player more than once simultaneously. As do_stuff finishes its work, it tells the parent process that it's done with the players, so that new tasks using those players can be processed.

import time
import multiprocessing
from Queue import Empty

listGames = [(0, 1), (0, 2), (0, 3), (1, 2), (1, 3), (2, 3)] 

def do_stuff(players):
    player1, player2 = players
    print("player1 {} player2 {}".format(player1, player2))
    time.sleep(5)
    # Imagine some other stuff happens here.
    print ("now done with {} and {}".format(player1, player2))
    q.put((player1, player2))

if __name__ == "__main__":
    q = multiprocessing.Queue()
    pool = multiprocessing.Pool()
    gamesSet = set(listGames)  # Convert to a set for efficiency reasons.
    running = set()  # This keeps track of players being processed.
    while gamesSet:
        to_remove = []
        for player in gamesSet:  
            if player[0] not in running and player[1] not in running:
                running.add(player[0])
                running.add(player[1])
                pool.apply_async(do_stuff, (player,))
                to_remove.append(player)
        for player in to_remove:
            gamesSet.remove(player)
        while True:
           # Find out if we're done processing any players.
           try:
               done = q.get_nowait()
               running.remove(done[0])
               running.remove(done[1])
           except Empty:
               break
    pool.close()
    pool.join()

Output:

dan@dantop2:~$ ./mult.py 
player1 0 player2 1
player1 2 player2 3
now done with 0 and 1
now done with 2 and 3
player1 1 player2 2
player1 0 player2 3
now done with 0 and 3
now done with 1 and 2
player1 1 player2 3
player1 0 player2 2
now done with 1 and 3
now done with 0 and 2