Getting multi-threading to work with a list using python pexpect

Question

I wrote a simple python pexpect script to ssh into a machine and perform a action. Now I need to do this action to multiple servers. I am using a list to hit all of the servers concurrently using multi-threading. My issue is due to everything being ran concurrently, each thread is running on the same server name. Is there a way to concurrently have each thread only run one of the listed servers?

    #! /usr/bin/python
#Test script

import pexpect
import pxssh
import threading
import datetime



currentdate = datetime.datetime.now()
easterndate = (datetime.datetime.now() + datetime.timedelta(0, 3600))

#list of servers
serverlist = ["025", "089"]

#server number
sn = 0


ssh_new_conn = 'Are you sure you want to continue connecting'

class ThreadClass(threading.Thread):
  def run(self):

    index = 0
    sn = serverlist[index]
    print sn
    username = '[a username]'
    password = '[a password]'
    hostname = '%(sn)s.[the rest of the host url]' % locals()
    command = "/usr/bin/ssh %(username)s@%(hostname)s " % locals()
    index = index + 1
    now = datetime.datetime.now()

    print command
    p = pexpect.spawn(command, timeout=360)

    ***do some other stuff****

for i in range(len(severlist)):
  t = ThreadClass()
  t.start()

[update] I may just trying doing this with a parent thread that calls the child thread and so forth....although it would be nice if multi-threading could work from a list or some sort of work queue.

Answer 1

The problem has nothing to do with "everything being ran concurrently". You're explicitly setting index = 0 at the start of the run function, so of course every thread works on index 0.

If you want each thread to deal with one server, just pass the index to each thread object:

class ThreadClass(threading.Thread):
    def __init__(self, index):
        super(ThreadClass, self).__init__()
        self.index = index
    def run(self):
        sn = serverlist[self.index]
        print sn
        # same code as before, minus the index = index + 1 bit

for i in range(len(severlist)):
    t = ThreadClass(i)
    t.start()

(Of course you'll probably want to use serverlist instead of severlist and fix the other errors that make it impossible for your code to work.)

Or, more simply, pass the sn itself:

class ThreadClass(threading.Thread):
    def __init__(self, sn):
        super(ThreadClass, self).__init__()
        self.sn = sn
    def run(self):
        print self.sn
        # same code as last version, but use self.sn instead of sn

for sn in severlist:
    t = ThreadClass(sn)
    t.start()

Alternatively, if you really want to use a global variable, just make it global, and put a lock around it:

index = 0
index_lock = threading.Lock()

class ThreadClass(threading.Thread):
    def run(self):
        global index, index_lock
        with index_lock:
            sn = serverlist[index]
            index += 1
        print sn
        # same code as first version

However, you might want to consider a much simpler design, with a pool or executor instead of an explicit worker thread and list of things to work on. For example:

def job(sn):
    print sn
    # same code as first version again

with concurrent.futures.ThreadPoolExecutor() as executor:
    executor.map(job, serverlist)

This will only run, say, 4 or 8 or some other good "magic number" of jobs concurrently. Which is usually what you want. But if you want exactly one thread per server, just pass max_workers=len(serverlist) to the ThreadPoolExecutor constructor.

Besides being a whole lot less code to read, write, get wrong, debug, etc., it also has more functionality—e.g., you can get results and/or exceptions from the servers back to the main thread.