Executing a binary file in a python script

Question

Edit: Using the snippet below I come across a few issues

import paramiko, threading

ssh=paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

def connectAndCMD(command):
    ssh.connect('127.0.0.1',22,'MY_USER','MY_SSH_PASS')
    stdin, stdout, stderr = ssh.exec_command(command)
    print stdout.read()

commandList=[
    "pwd",
    "whoami",
    "ls",
    "echo hello",
    "ping google.com", 
    "ifconfig | grep Bcast | awk {'print $2'}"
]
for command in commandList:
    print '[',command, '] command thread started'
    t=threading.Thread(target=connectAndCMD,args=(command,))
    t.start()

In the console I see: SSHException: No existing session Oddly enough, when I reduce the size of the list to just a few commands, reducing thread count, I am able to see the output of one of the ssh commands of a random thread but then the script hangs indefinitely. What is the problem and how do I go about rectifying this?

This is a mockup but in my actual program I have threads being made to run the same command with just different parameters/options

Answer 1

The problem is that you attempt to re-authenticate the ssh connection for each command. SSH uses a single transport connection (usually TCP but theoretically others) to multiplex multiple ssh-level channels which run things like commands, pipes and shells. SSHClient.connect will open a transport connection, authenticate the server (notice the missing host key policy - you used a rather insecure one) and authenticate the client (a password in your case).

After that, the client can open any number of ssh-level channels to do work. Each exec_command opens a channel, starts a shell on the server, runs a single command and closes the channel. But since these channels are multiplexed, they can run in parallel and are a good option for mulithreading.

import paramiko, threading

ssh=paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('127.0.0.1',22,'MY_USER','MY_SSH_PASS')

def connectAndCMD(command):
    stdin, stdout, stderr = ssh.exec_command(command)
    print stdout.read()

commandList=[
    "pwd",
    "whoami",
    "ls",
    "echo hello",
    "ping google.com", 
    "ifconfig | grep Bcast | awk {'print $2'}"
]
for command in commandList:
    print '[',command, '] command thread started'
    t=threading.Thread(target=connectAndCMD,args=(command,))
    t.start()

UPDATE

Since ssh servers limit the number of simultaneous commands you can use a thread pool to make sure you stay under the limit.

import paramiko
import multiprocessing.pool

ssh=paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('127.0.0.1',22,'MY_USER','MY_SSH_PASS')

def connectAndCMD(command):
    stdin, stdout, stderr = ssh.exec_command(command)
    return command, stdout.read(), stderr.read()

commandList=[
    "pwd",
    "whoami",
    "ls",
    "echo hello",
    "ping google.com", 
    "ifconfig | grep Bcast | awk {'print $2'}"
]

pool = multiprocessing.pool.ThreadPool(min(10, len(commandList))
for command, out, err in pool.map(connectAndCMD, commandList, chunksize=1):
    print((command, out, err))
pool.close()
pool.join()