4

I have a simple task which is scheduled by dask-scheduler and is running on a worker node.

My requirement is, I want to have the control to stop the task on demand as and when the user wants..

TheCodeCache
  • 820
  • 1
  • 7
  • 27

2 Answers2

2

You will have to build this into your task, perhaps by explicitly checking a distributed Variable object in a loop.

from dask.distributed import Variable

stop = Variable()
stop.set(False)

def my_task():
    while True:
        if stop.get():
            return

        else:
            # do stuff

future = client.submit(my_task)
# wait
stop.set(True)

You will need something explicit like this. Tasks are normally run in separate threads. As far as I know there is no way to interrupt a thread (though I would be happy to learn otherwise).

MRocklin
  • 55,641
  • 23
  • 163
  • 235
  • from dask.distributed import * import time def task_code(): out = None while True: if stop.get(): print("Running task stopped in between.!, returning..") return out else: print("Inside task_code()..", get_worker()) for _ in range(5): print("task is still in progress.!") time.sleep(_*2) out = "Hello World.!" print("now the task is over..!") return out – TheCodeCache Jan 06 '18 at 08:04
  • clienta = Client("192.168.1.2:8786") stop = Variable('xx', client=clienta) stop.set(False) futurea = clienta.submit(task_code) for _ in range(3): print("in client doing something else meanwhile..") time.sleep(_) stop.set(True) – TheCodeCache Jan 06 '18 at 08:05
  • it's still not working.. what if I am running an infinite loop inside the else: block.. the loop is still continue to work even I set the variable to True in order to stop the task.. I;ve pasted the sample code in the above comments in two parts. plz combine and see if you can provide further help.. Thanks.. – TheCodeCache Jan 06 '18 at 08:07
  • 1
    It is not possible to break a task actively running in a thread. This is a limitation of Python. You need to build in some sort of machinery yourself to periodically check some condition. – MRocklin Jan 06 '18 at 14:13
0

@MRocklin. thanks for your suggestion.. and here is the machinery that I've built around explicit stopping of the running/live task. Although the below code is not re-factored.. kindly trace the logic behind it.. Thanks - Manoranjan (I will mark your answer was really helpful..) :) keep doing good..

import os
import subprocess
from dask.distributed import Variable, Client 
from multiprocessing import Process, current_process
import time

global stop

def my_task(proc):
    print("my_task..")
    print("child proc::", proc)
    p = None
    childProcessCreated = False
    while True:
      print("stop.get()::", stop.get())
      if stop.get():
        print("event triggered for stopping the live task..")
        p.terminate()
        return 100
      else:
        if childProcessCreated == False:
          print("childProcessCreated::", childProcessCreated)
          p = subprocess.Popen("python sleep.py", shell=False)
          childProcessCreated = True
          print("subprocess p::", p, " type::", type(p))
      time.sleep(1)
    print("returnning with 20")
    return 20

if __name__ == '__main__':

    clienta = Client("192.168.1.2:8786")
    print("global declaration..")
    global stop
    stop = Variable("name-xx", client = clienta)
    stop.set(False)
    future = clienta.submit(my_task, 10)
    print("future::waiting for 4 sec..in client side", future)
    time.sleep(3)
    print("future after sleeping for sec", future)
    #print("result::", future.result())
    stop.set(True)
    print("future after stopping the child process::", future)
    print("child process should be stopped by now..")
    #print("future::", future)
    #print("future result::",future.result())

print("over.!")
TheCodeCache
  • 820
  • 1
  • 7
  • 27