I am trying to make use of the magic command %%bash --bg to run multiple bash commmands in the background asynchronously from a jupyter notebook session(note %%bash is an alias for %%script bash). I am doing this under the assumption that it will allow me to send subprocesses to different cpus. As a reduced example:
%%bash --bg --out out_variable --proc popen_instance
ls
Has the argumnts --out and --proc which return objects to the stdout and the popen instance as explained in the IPython documentation. When printing the variables in the next cells popen_instance prints:
<_AsyncIOProxy(<Process 1203497>)>
And out_variable outputs:
<_AsyncIOProxy(<StreamReader 453 bytes eof transport=<_UnixReadPipeTransport closed fd=64 closed>>)>
How can I use these to variable to track the progress of the background command that I have sent? The ideal for me would be for the ouput to be logged into a text file so I could open it up. And check how it is doing.
Reading around I found that using await out_variable.read() would work to check my output but as the name implies only after the subprocess has finished. I am intersted in tracking the progress of the process on the run.
