A couple of questions related to best practices with ipyparallel. I'm attempting use it to implement a Monte Carlo framework for a model that takes ~15 to run. The idea is to run N engines (via SLURM) and have a "master" process that queues all the required tasks asynchronously and busy-waits for completion, updating a sqlite db with the status of each run.
I would like to know when a task has been assigned to an engine so I can track its status in my database. I tried using the AsyncResult instances to get the msg_id and query the task database, but the "started" field isn't updated until the task completes.
It seems there should be a way to receive this notification, or at least to query the hub while the engine is working.
Also, must I do something to avoid engine heartbeat timeout during a long-running task? Is that the purpose of client.spin_thread()?
Thanks!