Lots of socket errors with celery eventlet tasks

Question

I'm getting a lot of "IOError: Socket closed" exceptions from amqplib.client_0_8.method_framing.read_method when running my celery workers with the --pool=eventlet option. I'm also seeing a lot of timeout exceptions from eventlet.hubs.hub.switch.

I'm using an async_manage.py script similar to the one at https://gist.github.com/821848, running the works like:

./async_manage.py celeryd_detach -E --pool=eventlet --concurrency=120 --logfile=<path>

Is this a known issue, or is there something wrong with my configuration or setup?

I'm running djcelery 2.2.4, Django 1.3, and eventlet 0.9.15.

I'm monkey patching every, but I'm not 100% certain that nothing is blocking. What's the best way to find out, and what can I do if they are? — Ben Dowling, Jun 20 '11 at 17:30
monkey patching only patches what eventlet knows how to patch, so you could still use libraries that are not covered by that. See here for example: http://unethicalblogger.com/2010/08/28/unclog-the-tubes-blocking-detection-in-eventlet.html — asksol, Jun 20 '11 at 23:40
I've enabled blocking detection and it occasionally blocks at "return self.connection.commit()" in django/db/backends/__init__.py - I'm using MySQL. Is there any way to avoid this? — Ben Dowling, Jun 21 '11 at 13:09
Discovered the undocumented MySQLdb option to monkey_patch - doing more testing now! — Ben Dowling, Jun 21 '11 at 13:31

score 6 · Accepted Answer · edited Oct 01 '21 at 23:12

The problem was a side effect of some code that was blocking. I managed to detect the blocking code using the eventlet option described in this article.

There were 2 places where blocking was occuring: DNS lookups, and MySQL database access. I managed to resolve the first by installing the dnspython package, and the second my using the undocumented MySQLdb option in eventlet:

import eventlet
eventlet.monkey_patch()
eventlet.monkey_patch(MySQLdb=True)

Lots of socket errors with celery eventlet tasks

1 Answers1