1

I am having an issue with SQLAlchemy in Celery tasks. I have one task that creates some objects, commits them to the database and then it adds another task with the IDs of these new database rows and the other task then fetches and works on those. However, the other task is not able to get the rows even though they are in the database.

To sketch it up:

@task(name="create-task")
def create_task():
    obj = SomeClass(column1="Something", column2="Else")
    session.add(obj)
    session.commit()

    consume_task.delay(obj.id)

@task(name="consume-task")
def consume_task(obj_id):
    obj = session.query(SomeClass).get(obj_id)
    # obj is None

These two tasks are run in two separate celery processes which initialises their own connection and session in the worker_process_init.connectsignal, but the session on the consume_task apparently is not able to fetch the rows from the database.

If I start the consume_task celery process after the create_task, then it works for those rows inserted before it was started. But I need them to run simultaneously.

Am I missing something completely obvious here about session behaviour? I thought that a SQLAlchemy session query would always query the actual database and not on some "snapshot" from when it was instanced."

user3419981
  • 21
  • 2
  • http://stackoverflow.com/a/17224993/1099876 – ndpu Apr 29 '14 at 12:01
  • @ndpu Thanks, but I already do that. This is between two celery processes using two seperate sessions. – user3419981 Apr 29 '14 at 12:08
  • Try [this](http://stackoverflow.com/a/21830134/1117099) will be fix your issue... – Syed Habib M Apr 29 '14 at 13:03
  • I'm having this same issue, take a look at this note from sqlalchemy http://sqlalchemy.readthedocs.org/en/rel_1_0/orm/session_basics.html?highlight=session%20basics "The bigger point is that you should not want to use the session with multiple concurrent threads. That would be like having everyone at a restaurant all eat from the same plate. The session is a local “workspace” that you use for a specific set of tasks; you don’t want to, or need to, share that session with other threads who are doing some other task." – Jorge Omar Vázquez Sep 20 '15 at 00:04

0 Answers0