Executing Python Script as Celery Task

Question

I am trying to run a python script as a celery task with Django. The issue I am having is that the task thinks it is complete as soon as the script begins running. I initially used subprocess.popen() in the tasks.py file, but realized this would mean the task would be complete as soon as the popen() command was issued. I modified my tasks.py code to call a function in my python script, which runs the script; however, this still executes as though the task is immediately complete. I am confused because in flower it says the task is complete, but in the celery log it is outputting the log data defined in the script I am running. I found the following related post. I believe I am following its suggestion to execute a python function from tasks.py.

tasks.py:

def exe(workDir, cancelRun):
    sys.path.append(workDir)
    import run

    if cancelRun=='True':
        task_id=exe.request.id
        revoke(task_id,terminate=True)
    else:
        run.runModel(workDir)
        task_id=exe.request.id
        return task_id

runModel function code:

def runModel(scendir):
    fullpath=scendir+'/run.py'
    os.chdir(scendir)
    p=Process(target=myMain,args=(scendir,))
    p.start()
    p.join()

What happens if you run the command (the one executed by your task) on your terminal? If it returns immediatelly it's possible that it's daemonizing itself, let's say via fork(). Another way of testing the cause is using the Django shell: python manage.py shell, then execute the necessary imports and call your functions manually. — jweyrich, Jul 10 '14 at 23:15
Executing from the command line does what I expect. It begins running a script, which typically takes several hours to complete. I also considered my logic for html may be wrong and I'm returning the page I want after the task has finished by mistake. — Jason Hawkins, Jul 10 '14 at 23:28
My issue was actually that I didn't understand the outputs from celery. I was looking at the 'worker' tab of flower, rather than 'tasks'. I also interpreted the logging in celery beginning at the start of my log file as a sign it was starting the process again. It turns out it was creating a second task (as expected and desired). It was good to verify in the command prompt everything was working. — Jason Hawkins, Jul 10 '14 at 23:36

score 1 · Accepted Answer · answered Jul 11 '14 at 00:43

What happens if you run the command (the one executed by your task) on your terminal? If it returns immediatelly it's possible that it's daemonizing itself, let's say via fork().

Another way of testing the cause is using the Django shell: python manage.py shell, and then execute the necessary imports to call your functions manually.

Executing Python Script as Celery Task

1 Answers1