2

I have a Django application that use Celery to create async tasks. Some of these tasks live within the Django project and other live on a remote worker with its own code base.

I currently use the django-celery-results package to store the results of task calls within the Django database so that I can easily query the status of calls using the Django ORM. This works fine when I call my "local" tasks, but it does not seem to work as well when I call remote tasks.

For example:

app.send_task("django.foo")  # Return status SUCCESS and store result in DB
app.send_task("remote.bar")  # Stuck in PENDING and never create result in DB

By reading the Celery Docs I found out that tasks can be stuck in PENDING if the client and worker doesn't use the same CELERY_RESULT_BACKEND setting. In this case I can't use the django-db backend on my remote worker, since it's not a Django application.

So in this case... How do I store my results when doing remote calls in this manner?

Note that in the case of remote.bar, I confirm that the remote worker receives the message and executes the method. Its just that my client (Django App) doesn't receive the response.

Marcus Lind
  • 10,374
  • 7
  • 58
  • 112
  • Use Django Background Task Instead Of Celery Because Its Easy Take a look at my answer here: https://stackoverflow.com/questions/54225303/signal-django-to-run-a-task/54225921#54225921 – Ahtisham Jan 21 '19 at 16:19
  • So you don't want to use database as result backend. You have plenty of other options, you can use redis: http://docs.celeryproject.org/en/latest/userguide/configuration.html#task-result-backend-settings – zeynel Jan 21 '19 at 16:20

2 Answers2

0

n.b., if different celery workers are using different result backends, you can configure the app instance on the fly.

from celery.app import Celery
app = Celery(broker_url='my_broker_url', result_backend='non-django-orm backend')
result = app.send_task('remote.bar')
# you can check status here just fine.

If you want to support such a configuration, you're going to need multiple workers since each worker can only connect and store results to a single backend.

2ps
  • 15,099
  • 2
  • 27
  • 47
  • I've done custom implementations of amqp workers using the Pika library in the past. In those cases I've send messages, awaited for responses on the client side and then store it in some result backend on the client side. Isn't this how Celery result backends also work? Shouldn't I be able to retrieve an amqp response and then store it in the Django DB so I can use signals and other tools to create callbacks for when tasks are completed or failed? – Marcus Lind Jan 22 '19 at 02:43
  • This is not how celery works. The way that celery works is that celery workers consume tasks sent via the broker and then store the result of the task execution into the result backend. Clients that care about the result or outcome of any particular task poll the result backend to see if their task is ready. The reason for this is that celery is meant for distributed multiprocessing, so celery assumes that clients are not be available at the time the task completes and the result needs to be stored. – 2ps Jan 22 '19 at 08:36
0

As the other answer already stated, you can set up the result_backend. I had to specify it with Celery(result_backends=....) as the SQL address URL I've set in Django settings was getting ignored. The problem with this approach, however, is that you need to provide a direct access to the database, which makes things simple, but not secure if your celery workers need to be in a different network.

If the security is a concern, you will have to create APIs to update the results from the client. Here is more explanation on such approach.

REST API or "direct" database access for remote Celery/Django workers?

Forethinker
  • 3,548
  • 4
  • 27
  • 48