Using eventlet.corolocal with Celery

Question

Does anyone have experience using eventlet.corolocal, especially with celery (with eventlets for workers)?

If so, could you shed some light on what's wrong with the below code sample?

...
from eventlet.corolocal import local

...

ev_local = local()

@app.task
def dummy_task(self, a):
  if hasattr(ev_local, a):
    ev_local.a += a
  else:
    ev_local.a = a
  print ev_local.a

if __name__ == '__main__':
  app.start()

If I now start a celery worker with concurrency, say 5,

celery multi start 1 -A process_mss -l info -P eventlet -c 5 --verbose

and call the task (dummy_task) 20 times -

...

for i in xrange(20):
  dummy_task.delay(1)

I see that the 5 eventlets process 4 tasks each (using id returned eventlet.corolocal.get_ident()). But, the eventlets always find ev_local without knowledge of the variable/attr 'a'. So the print statement in dummy_task(...) always prints 1.

Any pointers on what could be going wrong?

score 0 · Answer 1 · answered Aug 09 '15 at 15:24

0

There's a small typo here: if hasattr(ev_local, a):

Second argument to hasattr must be string: hasattr(ev_local, 'a').

answered Aug 09 '15 at 15:24

temoto

5,394
3
34
50

score 0 · Answer 2 · answered Dec 06 '16 at 20:50

Eventlet doesn't guarantee you will receive the same greenthread. In fact, each task may run in a new greenthread.

The greenthread id (eventlet.corolocal.get_ident()) is the memory address of a Python object (I think the greenthread). So when a new greenthread is used to execute the celery task, the thread local storage will "disappear." If you happen to get lucky and a celery task reuses a greenthread, then the information in the thread local storage will reappear.

You can work around it using the prefork threading model.

To date, I have yet to solve this problem.

Using eventlet.corolocal with Celery

2 Answers2