Celery worker variable sharing & initialization in bootstep

Question

I have a question regarding process shared variables in python 3.4 and celery 4.0.2. I already read a post (Celery worker variable sharing issues) where the goal of the poster was to not share the variable.

I currently have the exact opposite of this problem: I want to share a variable across all subprocesses of a single worker.

My situation is as follows:

i'm using prefork on my celery worker (running under ubuntu)
i created a bootstep where i retrieve data from a configuration server and store the dict in a global variable thats placed in its own module. the bootstep is a worker bootstep and its only dependency is the timer. i do the (blocking) request in the bootsteps start method. i can see that it receives the correct data
sometimes it "just works" i see the global variable from my worker subprocesses. sometimes it does not work, the variable is initialized but the dict inside it where the data should be is empty
i already did some debugging and found out that in fact the ids of the objects that should hold my config dict are the same (which suggests that the process was copied after the configuration variable was first accessed

config.py in module config

class MyGlobalConfig(object):
    def __init__(self):
        self.data = {}

    def get(key):
        return self.data[key]

global_config = MyGlobalConfig()

bootstep.py

import config

...
global_config.data = response_data
...

some.py accessed from a task

import config

...
global_config.get(key) # NoneType cause global_config.data is empty
...

i have no idea why it works sometimes and sometimes not. from what i've seen celery forks its subprocesses after the worker bootsteps have finished - so in theory my data should be there.

apart from these startup issues it is totally possible that i get config updates during the lifetime of the worker and i need to distribute this to all subprocesses as well.

any ideas whats the best way do this in celery? everything i found so far is either worker centric or utilizes the broker. since it should only be for the local workers processe i don't want to use anything that could affect other non local workers or utilizes the broker ...

Celery worker variable sharing & initialization in bootstep

0 Answers0