Can you pass an object to a celery task by its memory id using ctypes?

Question

I want to pass a object of my own class to a celery task. I am not using Django, this is my own custom class which is not serializable.

After researching, I got the idea to pass the object memory id as an argument and then get the object from the id as per this answer:

tasks.py

import ctypes

@app.task
def my_task(obj_memory_id):
    my_obj = ctypes.cast(obj_memory_id, ctypes.py_object).value
    my_obj.my_method()

main.py

def main():
    obj = MyClass()
    obj_memory_id = id(obj)
    my_task.delay(obj_memory_id)

Now, this works when I perform it outside celery. But when I do it with celery, I get:

billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).

Why is that and how can I achieve my goal?

More info: I am not instantiating the class inside the Celery task, because this class instantiates very slow (1-2 seconds). For my purposes, even a delay of 1 second is a lot. I want to have its instance ready in advance, and when I need to call its method, to do it immediately.

Never, **ever** do this in production code: `ctypes.cast(obj_memory_id, ctypes.py_object).value`. — juanpa.arrivillaga, Oct 30 '19 at 21:26
Because it relies on an implementation detail, (that the `id` is the memory address of the PyObject header), and an unsafe cast. In any case, your celary process won't be able to make sense of the memory address from a separate process like that, see: https://stackoverflow.com/questions/4352668/can-i-pass-an-object-to-another-process-just-passing-its-pointer-to-a-shared-me — juanpa.arrivillaga, Oct 30 '19 at 21:29
@juanpa.arrivillaga Thanks. So are you aware of any way to pass unserializable object to a celery task? — Nikolay Shindarov, Oct 30 '19 at 21:37

Markus Unterwaditzer · Accepted Answer · 2019-10-30T21:30:09.663

3

The celery worker runs in a different process, potentially on a different machine. It is unlikely to share memory with the process that spawns the task. You are passing a random pointer to the task, that task dereferences it and you get garbage.

You need to make your object serializable in some way if you want to use Celery in a useful way. Either by making it pickleable or something better.

edited Oct 30 '19 at 21:30

answered Oct 30 '19 at 21:27

Markus Unterwaditzer

7,992
32
60

Is it impossible if it runs on the same machine (including the broker)? – Nikolay Shindarov Oct 30 '19 at 21:29
2

@NikolayShindarov you can do that but why do you want to run Celery at all then? – Markus Unterwaditzer Oct 30 '19 at 21:30
Let's say I have something like observer pattern. But when from the observable I call the update method on the observer, I want to call it asynchronously in the background and keep running the observable without waiting for the observer's execution. So is there any way to achieve what I want? – Nikolay Shindarov Oct 30 '19 at 21:36
@NikolayShindarov just use a thread then, right? Celery is useless if you can't serialize your task arguments. – Markus Unterwaditzer Oct 30 '19 at 21:37
I might look into threads, thanks. But I still don't understand why Celery is useless if the task arguments are not serializable? – Nikolay Shindarov Oct 30 '19 at 21:39

Can you pass an object to a celery task by its memory id using ctypes?

1 Answers1