4

I have written an API with Django which purpose is to operate as a bridge between a website back-end and external services we use, so that the website doesn't have to handle many requests to external APIs (CRM, calendar events, email providers etc.).

The API mainly polls other services, parses the results and forwards them to the website backend.

I initially went for a Celery-based task queue, as it seemed to me like the right tool to offload that processing to another instance, but I'm starting to think it doesn't really fit the purpose.

As the website expects synchronous responses, my code contains a lot of :

results = my_task.delay().get()

or

results = chain(fetch_results.s(), parse_results.s()).delay().get()

Which doesn't feel like the proper way to use Celery tasks.

It is efficient when pulling dozens of requests and processing the results in parallel - a periodic refresh task for example - but adds a lot of overhead for simple requests (fetch - parse - forward), which represent most of the traffic.

Should I go full synchronous for those "simple requests" and keep Celery tasks for specific scenarios ? Is there an alternative design (maybe involving asyncio) that would better suit the purpose of my API ?

Using Django, Celery (w/ Amazon SQS) on an EBS EC2 instance.

Rémi Héneault
  • 383
  • 3
  • 12
  • 1
    Partial answer : https://stackoverflow.com/questions/42657786/django-should-external-api-requests-always-be-made-through-a-task-handler-e-g?rq=1 – Rémi Héneault Feb 28 '19 at 11:49

1 Answers1

6

You could consider using Gevent with your Django webserver to allow it to operate efficiently for the "simple requests" you've mentioned without being blocked. If you proceed with this approach, be sure to pool database connections with PgBouncer or Pgpool-II or a Python library since each greenlet will make its own connection.

Once you've implemented that, it's possible to also use Gevent instead of Celery to handle asynchronous processing by joining on multiple Greenlets that each make an external API request, rather than incur the overhead of passing messages to an external celery worker.

Your implementation is similar to what we've done at Kloudless, which provides a single API to access multiple other APIs, including CRM, calendar, storage, etc.

vinod
  • 2,358
  • 19
  • 26
  • 1
    Thank you for your answer! Would you recommend dropping Celery entirely and go for Gevent or using both depending on the task? How did you choose between Gevent and Eventlet ? Finally, how do you monitor your greenlets ? – Rémi Héneault Mar 01 '19 at 09:47
  • 2
    Certainly does depend on the task at hand: we actually chose Celery for most of our processing since we also wanted to launch periodic background jobs, and we wanted to minimize the load on the webservers while multiple background API requests were in progress so we could scale those workers separately. I don't have as much experience with monitoring greenlets as a result, but we do use those heavily for simpler asynchronous tasks within the primary API server. We didn't consider eventlet much during the development process. – vinod Mar 01 '19 at 13:23