IPython.parallel's load balanced interface has been great. I used it to run complicated scientific computations on a small cluster environment. Several houndred problems, each taking 5~10 minutes to complete on a PC, were sent to the cluster. I had no problem getting the results back until I started submitting more strenuous tasks to the cluster. Some engines were unregistered even though the task assigned completed successfully. I use "ipcontroller --ping=120000" to circumvent the problem, but IPython now tells me "Task farming is disabled". What is causing the problem and what should I do?
By the way, I use ssh mode on the cluster.