dnspython and 'Lookup timed out'

Asked Mar 29 '13 at 15:22

Active Mar 29 '13 at 19:05

Viewed 646 times

I'm using Django, Celery, Eventlet and dnspython to asynchronous parse about 500 rss feeds.

Using dnspython causes 'lookup timed out' error when I try to parse more than 20 feeds at the same time. When I uninstall dnspython all works great, but I lose some time because dns lookups are blocking celery pool. Do you have any ideas how can it be fixed?

this is my celery task code:

import eventlet
feedparser = eventlet.import_patched('feedparser')
from celery import group

@task(ignore_result=True)
def update_feeds():
    group(update_feed.s(feed) for feed in Feed.objects.filter(active=True)).apply_async()

@task(ignore_result=True)
def update_feed(feed):
    parsed_feed = feedparser.parse(feed.feed_url, etag=feed.etag, modified=feed.modified)
    # It fails when I have dnspython installed returning <urlopen error (-3, 'Lookup timed out')> error

I'm using Ubuntu 12.04 LTS

edited Mar 29 '13 at 19:05

asked Mar 29 '13 at 15:22

Patrykw

Can you reproduce the issue without Eventlet, using threads? – temoto Mar 30 '13 at 07:49
full discussion https://github.com/rthalley/dnspython/issues/27 – Paolo Aug 22 '21 at 01:15

dnspython and 'Lookup timed out'

0 Answers0