Retrieve engine id in function

Question

I am using ipython parallel to schedule a large number of job using load_balanced_view. Each job uses subprocess.Popen to run a code and retrieve stdout and stderr. Then I want to save them into a log file.

This is an example of the structure of the code that I'm running:

import subprocess as sp

def make_inifile(c):
    ...
def run_exe(exe, inifile):
    def_Popen_kwargs = {'stdout': sp.PIPE, 'stderr': sp.PIPE, 
                        'universal_newlines': True}
    pexe = sp.Popen([exe, inifile], **def_Popen_kwargs)
    (stdout, stderr) = pexe.communicate()
    with open('logfile.log', 'a') as f:
        f.write(stdout)
        f.write(stderr)

rc = Client()
lbv = rc.load_balanced_view()
rc[:].execute("import subprocess as sp", block=True)

exe = "/path/to/exe"
for c in cases:
    inifile = make_inifile(c)
    lbv.apply(exe, inifile)

lbv.wait()

As I'll be using more than 1 processor, the log file will look like a mess, in the best case. A solution might be to lock the file, such that only a process at a time can write to it. This should be feasible, but look a bit overkill to me.

A better solution might be to open a log file for each engine, using the engine id in the file name. Something like this: "logfile_{}.log".format(id)

So the question is: is there a way to retrieve the engine id from within run_exe?

Not what you're asking, but another possible solution would be to write the strings to a queue and have another process write them to a single file. [This link](http://ipython.org/ipython-doc/dev/development/parallel_messages.html) may be useful — loopbackbee, Nov 13 '13 at 14:50
Ehm. Thanks for the extra way. I think I understand what you mean, I just have no idea about how to do it. I'll try to read and understand the link. (anyway seems a bit overkill for what I need now, but I like learning new stuff :D) — Francesco Montesano, Nov 13 '13 at 15:08

score 2 · Accepted Answer · answered Nov 13 '13 at 14:49

2

Either push client id to client at startup with a dview or use os.getpid() ?

answered Nov 13 '13 at 14:49

Matt

27,170
6
80
74

1

Thanks. Two questions: 1) with push client id you mean this: `rc[i].push({'id': rc.ids[i]})`? 2) is every engine and independent process with its own fixed id? – Francesco Montesano Nov 13 '13 at 15:00
2

1) Yes, 2) IIUC only the controller know ID; so you have to push a value, unless using MPI but then you need to map between MPI id and IPython IDs. Yes each engine is a separate process. Yes id is fixed (for the controller) as long as the process does not die. os.getpid() is os specific and follow os rules. Usually strictly increasing until reboot, so could be duplicate if engine on many machines. – Matt Nov 13 '13 at 18:54

Retrieve engine id in function

1 Answers1