DB changes made in a Celery task not visible in the main thread

Question

When updating or creating a new object in a Celery task, reading the new data in the main thread fails. After the update / create I issue

db.session.commit()

From cmd it seems that the data was updated in the DB, but maybe something with the session is still getting the old data. When I update some objects in the same thread it works ok.

I'm guessing that you're using MySQL? Do you ever commit or rollback the transaction running in your main thread? If not, you've found the effects of transaction isolation. — Ilja Everilä, Oct 03 '19 at 05:36
yes using MySQL. Do you mean I need to add db.session.commit() both in Celery task code and in the main thread? — Adi Ep, Oct 03 '19 at 06:02
Pretty much. The default transaction isolation level in MySQL is REPEATABLE READ, which is pretty much what it sounds like. You'll have to end the transaction in order to create a new snapshot from current data, either by committing or rolling back. — Ilja Everilä, Oct 03 '19 at 06:24
Could you please provide a [mcve] in the question? It'd make it a lot clearer what's happening and why (for future readers that are looking for a solution to the same issue). — Ilja Everilä, Oct 03 '19 at 06:35

score 0 · Answer 1 · edited Oct 03 '19 at 12:27

actually I dont sure why we need to use commit command on Celery task to end the transaction and then again on the main thread. below there is working example:

this celery task is running each 7 seconds.

@scheduler.task()
def run_task():
    task = Tasks.get_task("task1")
    task.update_task({'status': "succeed"})
    db.session.commit()


def test_update_task(self):
    task = create_task("task1", status="active")

    log.warning("task {} status before change:{}".format(task.id, task.status))

    time.sleep(10)

    # Here the celery task will update task status from "active" to "succeed"
    db.session.commit()
    log.warning("test {} task status after Celery change:{}".format(task.id, task.status))

DB changes made in a Celery task not visible in the main thread

1 Answers1