Using Django for a small app that executes jobs in the background (mainly cron style, but also user initiated as well).
disclaimer: this is my first encounter with celeryd, setup so far is from docs and examples
I'm using Django 1.3, celeryd 2.5, django-celery 2.5.1, and sqlite3.
I'm still in the testing phase, so I'm not invoking a real task, but just printing out messages.
I have a cron style task:
@periodic_task(run_every=crontab(hour="*", minute="*", day_of_week="*"))
def test():
print "firing test task"And another task that will be user initiated:
@task(name="myapp.tasks.user_task")
def user_task(country):
print "performing task for: "+countryThe cron style task will only be run once per day, while the other will be executed sparingly
(from 0 to 30'ish times per day on average)
I'm using sqlite as my backend and my celery settings are:
# Celery
INSTALLED_APPS += ("djcelery",)
import djcelery
djcelery.setup_loader()
BROKER_URL = "django://"
BROKER_POOL_LIMIT = None
CELERYD_CONCURRENCY = 1The problem I'm running into, which shouldn't happen, but will because it can, is if the user initiated task user_task is called a few times in rapid succession celery shuts down with the following:
Unrecoverable error: TransactionManagementError('Transaction managed block ended with pending COMMIT/ROLLBACK',)
further down the stack
TransactionManagementError("Transaction managed block ended with "
TransactionManagementError: Transaction managed block ended with pending COMMIT/ROLLBACK
According to celerd docs, using django-celery w/sqlite I can only process one task at a time, which is perfect for my case, but seems like multiple tasks are stepping on top of eachother.
I've set the concurrency level to 1, and pool limit to None, to try and help out, but I'm still getting the same error.
Am I missing something here with celery? Doing something wrong? Is there a better approach to what I'm trying to accomplish?
