1

I have a project with lots of Celery tasks, and one of the tasks must be executed only one at a time (it is a request to a 3rd party API which disallows several concurrent connections).

I can achieve this by starting a separate celery process with a separate queue and a concurrency of 1.

Regular celery process:

celery -A sourcery worker -Q default -c 4

A separate single-worker process:

celery -A sourcery worker -Q separate_queue -c 1

But I am on Heroku, and I will be billed doubly for spinning up two processes instead of one. So, is there a way to achieve it with a single Celery process?

kurtgn
  • 8,140
  • 13
  • 55
  • 91

1 Answers1

0

There is currently no way to do this.

I had a similar issue where I had a 3rd party API which only allowed one concurrent connection. I ended up running two separate dynos on Heroku.

Somebody did request this feature, but it has not been implemented: https://github.com/celery/celery/issues/1599

Edit:

There is something called celery-multi that could be worth looking into:

Celery - run different workers on one server

I'm not sure it's Heroku compatible though, let me know if you try!

Daniel
  • 4,918
  • 4
  • 33
  • 34