Share memory areas between celery workers on one machine

Question

I want to share small pieces of informations between my worker nodes (for example cached authorization tokens, statistics, ...) in celery.

If I create a global inside my tasks-file it's unique per worker (My workers are processes and have a life-time of 1 task/execution).

What is the best practice? Should I save the state externally (DB), create an old-fashioned shared memory (could be difficult because of the different pool implementations in celery)?

Thanks in advance!

What about cache like memcached, or key-value stores like Redis? — demalexx, Mar 05 '12 at 11:40
That would be a way, but I hoped that I can find a way without an additional dependency. — Gregor, Mar 05 '12 at 11:42
It's tiny dependency but it'll solve some problems you may encounter implementing own inter process communication. — demalexx, Mar 05 '12 at 13:17

score 9 · Accepted Answer · answered Mar 05 '12 at 13:34

9

I finally found a decent solution - core python multiprocessing-Manager:

from multiprocessing import Manager
manag = Manager()
serviceLock = manag.Lock()
serviceStatusDict = manag.dict()

This dict can be accessed from every process, it's synchronized, but you have to use a lock when accessing it concurrently (like in every other shared memory implementation).

answered Mar 05 '12 at 13:34

Gregor

4,306
1
22
37

Hi Gregor, I'd be happy to have your opinion on this issue I'm facing: http://stackoverflow.com/questions/26088868/asynchronous-task-queue-processing-of-in-memory-data-stucture-in-django Do you think your solution would be suitable for my problem there? Thanks a lot – jhagege Sep 29 '14 at 03:51
Hi cyberjoac, in general this could work, but you need to try out if Django and Celery live in the same process group (which I doubt). Let me know if it works, I'm interested! – Gregor Oct 03 '14 at 11:10
2

Where do you init this Manager? – Antonio Beamud Dec 23 '14 at 09:53
It basically does not matter, @AntonioBeamud - it needs to be in a module which is loaded for every process. – Gregor Dec 30 '14 at 11:12
I have main.py. I have generatereport.py which generates reports and I added the dictionary here. Thanks to you, all the workers have the access to the same dictionary. But I cant access this dictionary in my mian.py. How to achieve this. – Random Oct 31 '17 at 20:18
Its not working for me. My celery workers exit with -1 – Parikshit Chalke Aug 22 '19 at 10:02

Share memory areas between celery workers on one machine

1 Answers1

Linked