0

I am trying to create a python script that calls 5 other python scripts to run simultaneously while passing in an array and then each of the 5 scripts perform an operation on that array and return a different array to the initial script.

The initial script then realize when the 5 have returned values and then performs operations on these 5 arrays.

I think the solution is something like os.system(./script1.py arg1), os.system(./script2.py arg2) but I'm unsure of how to proceed.

Tonechas
  • 13,398
  • 16
  • 46
  • 80
Jai Pancholi
  • 81
  • 1
  • 4
  • You should not use `os.system` if you want to run the five scripts simultaneously. Use the `subprocess` module. – kindall Jan 07 '17 at 17:55

1 Answers1

0

You can use a thread pool to run all of the commands in parallel. I also changed over to the subprocess module which grabs program outputs:

import multiprocessing.pool
import subprocess as subp

def worker(script):
    proc = subp.Popen(script, shell=True, stdout=subp.PIPE, stderr=subp.PIPE)
    out, err = proc.communicate()
    return script, out, err, proc.returncode

scripts = ['./script1.py arg1', './script2.py arg2']
pool = multiprocessing.pool.ThreadPool(len(scripts))
for script, out, err, returncode in pool.map(worker, scripts):
    do your magic
pool.close()
pool.join()
tdelaney
  • 73,364
  • 6
  • 83
  • 116
  • this looks kind of like what i'm after, but im unclear of what the for script, out, .. do your magic is for. Also where are the outputs of the different scripts returned? – Jai Pancholi Jan 07 '17 at 18:26
  • The pool creates one thread per item in `scripts`. `pool.map` sends each script and the worker function to the threads and then acts as an iterator to pass the worker return values to the main program. So, the for loop gets the result of each script in turn. Depending on how you want to process the data, you could juggle what is returned and how its assembled. For instance, `worker` could just return `out` and you could get a list of outputs as `outputs = list(pool.map(worker, scripts))`. – tdelaney Jan 07 '17 at 18:48
  • wow. thanks a lot. this works beautifully. Is there a way I can consider the output of each of the scripts separant from what they print? i.e. in pool.map is there a return value as well as an out value? – Jai Pancholi Jan 08 '17 at 13:11
  • The [Popen.communicate](https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate) docs explain it the best. This example returns the program's `stdout`, `stderr` (what the program normally prints) and `returncode` (a single integer) and that's what you iterate though in pool.map's return. Give it a try by printing the variables in the for loop to see what's going on. – tdelaney Jan 08 '17 at 20:39