10

I am using MySql as result backend for celery. I want to store program arguments in DB too.

For example: 

add.apply_async(args=[num1, num2, user]

In this case I want to store user argument in DB, so that I can query later.

Currently, I return arguments provided which is stored in DB.

def add(num1, num2, user):
    return [num1+num2, user]

However when the task is running state, user is not inserted and I am unable to query it in DB. Is there any solution/hack for this?

user6035379
  • 181
  • 3
  • 16
Rajs123
  • 663
  • 13
  • 30

3 Answers3

1

Are you only using MySql as a result backend, or are you using it for the queue as well (which I wouldn't recommend)? If you're using it for a queue, the args should be in the database as soon as the task has been sent. Otherwise, the task result can't be stored until after the task has finished.

If you need the arguments to be query-able while the task is running, you'll need to manually insert them into a DB table at the start of your task. If you want them to be query-able before the task starts, you'll need to insert them into a DB table right before calling apply_async.

Seán Hayes
  • 4,060
  • 4
  • 33
  • 48
  • Thanks! There one problem through. While submitting each command, a new DB connection would open and when multiple cmds are run there might be an issue. I was looking for some celery soln which might handle DB connections as well. – Rajs123 Nov 11 '16 at 10:19
  • I'd recommend using a pool of open DB connections. – Seán Hayes Nov 11 '16 at 21:56
0

Assuming you are able to connect and select to the db, you may want to separate a function that query parameters and a function callback to store your results in db.

Add() will operate results, and store_callback() will store them in db when ready. So if parameters are ready, your code can go on next task and store result (user) when done.

Something like:

def store_callback(result):
     sql_insert = 'INSERT INTO your_table VALUES(?, ?,)'
     curs.execute(sql_insert, result)  #result is a list passed from add()

def add(num1, num2, user):
    return [num1+num2, user]


# check parameters are present in db:
curs.execute("SELECT * FROM your_table WHERE user = ?", [_user])
user_exists = curs.fetchone()
#
if user_exists:
    add.apply_async( (num1, num2, user, ) , link=store_callback.s() )

you can even link add() to another task.

user305883
  • 1,635
  • 2
  • 24
  • 48
  • callback will store results after job has been executed. I need to query the job status, cmd and arguments (with given task-id) during execution. – Rajs123 Nov 11 '16 at 10:19
0

You need queue,each user build a queue for himself, celery tasks get arguments from the user queue.

queue.get(timeout=10)# if nothing got then retry util get the arguments
blueboy
  • 44
  • 1
  • 11