django-nose unit testing a celery task ... missing database data

Question

I'm writing unit tests for a celery task using django-nose. It's fairly typical; a blank test database (REUSE_DB=0) that is pre-populated via a fixture at test time.

The problem I have is that even though the TestCase is loading the fixture and I can access the objects from the test method, the same query fails when executed within an async celery task.

I've checked that the settings.DATABASES["default"]["name"] are the same both in the test method and the task under test. I've also validated the that the task that's under test behaves correctly when invoked as a regular method call.

And that's about where I'm out of ideas.

Here's a sample:

class MyTest(TestCase):
    fixtures = ['test_data.json']

    def setUp(self):
        settings.CELERY_ALWAYS_EAGER = True # seems to be required; if not I get socket errors for Rabbit
        settings.CELERY_EAGER_PROPAGATES_EXCEPTIONS = True # exposes errors in the code under test.

    def test_city(self):
        self.assertIsNotNone(City.objects.get(name='brisbane'))
        myTask.delay(city_name='brisbane').get()
        # The following works fine: myTask('brisbane')

from celery.task import task

@task()
def myTask(city_name):
    c = City.objects.count() # gives 0
    my_city = City.objects.get(name=city_name) # raises DoesNotExist exception
    return

What database and what version of celery/django-celery are you using? — Mark Lavin, Jun 08 '12 at 20:13

score 2 · Accepted Answer · answered Jun 08 '12 at 20:20

2

This sounds a lot like a bug in django-celery 2.5 which was fixed in 2.5.2: https://github.com/celery/django-celery/pull/116

The brief description of the bug is that the django-celery loader was closing the DB connection prior to executing the task even eager tasks. Since the tests run inside a transaction the new connection for the task execution can't see the data created in the setUp.

answered Jun 08 '12 at 20:20

Mark Lavin

24,664
5
76
70

1

Nicely spotted. I was running 2.5.1... :-( – Dwight Gunning Jun 09 '12 at 20:29
I'm quite familiar with this bug and its frustrations. I was the one who submitted the pull request :) – Mark Lavin Jun 10 '12 at 00:00
Wondering if you ever had similar problems with the results not being stored under tests? Right now I can get the eager tasks to update the database but the result record (database result store) doesn't seem to be stored. I wonder if it's a side-effect of the transactions... is there any doco I should be reading/understanding? – Dwight Gunning Jun 11 '12 at 19:33
I typically ignore my task results (sending emails, fetching RSS feeds) so I haven't seen the problem with the results but it wouldn't surprise me if it was a similar problem with the DB connection. Not really a lot of documentation on the inner works of the django-celery loader so I typically just go straight to the source. – Mark Lavin Jun 11 '12 at 19:53
Found there's already a pull request... https://github.com/celery/django-celery/pull/104 – Dwight Gunning Jun 11 '12 at 20:37

django-nose unit testing a celery task ... missing database data

1 Answers1