Fair scheduling of task execution time between web service users

Question

Suppose we have the following web service. The main function is doing screenshots for the given website URL. There is REST API and user interface for entering URLs. For each new URL is a task in Celery is created. For frontend UI is important that screens for some URL will follow in a reasonable time, like 10 seconds.

Now a user, intensionally or by a software error, enters few hundreds URLs. This bloats task queue and other users must wait until all those tasks will be done.

So the request here is to:

1. Running tasks in some fair order. The simplest solution is to run one task for each user in one time. Like: user1 task, user2 task, user1 task, user2 task, and so on.
2. Having some priorities on tasks. Like tasks of priority 1 is always done before tasks of priority 2.

Currently, we utilize our handcrafted module. It stores tasks in Redis and pushes them in fair order to Celery. To not depend on Celery ordering it pushes only as many tasks as there are free Celery workers available, and checking Celery queue for free workers every 100 milliseconds.

Are there any libraries or services which meet my requirements?

Answer 1

• How many tasks do you have?
• How many users you have?

Sounds like you need rate-limiting mechanism in your webserver per user. For your question, there are serval options:

1. you can use celery router and assign different tasks for different queues (and then consume from those queues by different workers.
2. Celery support tasks priority, you can read about it here.
3. You can rate-limit per task in Celery - again, depends on your usage.

EDIT: @uhbif19 I described those features since you asked for them - you wanted a way to achieve priority and you send tasks with a specific priority.

In your current architecture you might want to decrease priority to abusers and avoid starvation of other users.

A better way to tackle this problem IMO is to add a rate-limiting mechanism in the gateway and ensure that a single user won't be able to abuse the system and make starvation for all others.

Good luck!