Celery: Corrupted state payload. Possible causes? Decorated __call__() method the cause?

Question

I'm trying to diagnose this but want to throw the question out there in case anyone has seen anything like this.

Essentially I'm seeing AsyncResult(task_id), return a result which has a state of None.

Eeerie.

I'm using an rpc: backend.

So I've drilled down into the Celery code and the RabbitMQ queues and from both angles I can see that the state message looks like this:

dict: {'task_id': 'STARTED', 'status': None, 'result': {'pid': 24113}, 'traceback': None, 'children': []}

Clearly corrupt. STARTED should be under status.

I even have some clue as to what is causing this, I have just to prove it and find the evidence, and how to fix it. It's laborious to say the least and so I'd love to a) speed it up with nay experience on offer and b) put it down on record here if I do solve it, for posterity's sake (in case anyone else runs across this).

This could (maybe) relate to my decorating the task.call() method. But I'm stuck figuring how.

A gem of an insight would be where in Celery is the state message sent to the backed RPC queue? It seems to be compiling it wrongly. Thus far I've traced it as far as sending a task-started event to the celerev exchange (for events I imagine) and it may be this event triggers an update of status to be sent.

I'll continue delving. But while I sleep, perhaps someone else has a clue?

Answer 1

Ouch, found the issue. And not sure when this crept into the codebase!

The hours I spent tracing code to find this are almost embarrassing as in the end, and total blind human error. The code that breaks is:

self.update_state(state, meta=meta)

What works is:

self.update_state(state=state, meta=meta)

and that's simply because Task.update_state() has this signature:

def update_state(self, task_id=None, state=None, meta=None, **kwargs):

that us, task_id us the default. At some point for some reason working code on was altered. Hmmmm, the story of how and when is not clear alas (part of a rather major refactor between commits).

Frustrating is, in retrospect this was the first place I should have looked. Reson I failed in that methinks is because the first Celery state updates (standard Celery state update (STARTED) was broken and that isn't generated by the line above I fixed. Mystery to me.