How to pause and resume RQ task

Question

I want to enqueue the RQ task, to complete a task, suppose it takes around 10 minutes, Now i want to pause any task if it continues for 5 minutes and start another task and after another 5 minutes and i want to pause the second task and start the 3rd one. Eventually, i plan to resume all the tasks and finish them

r = redis.Redis(host='localhost')
q = Queue(connection=r)

app = FastAPI()
@app.get("/add")
async def add_task(url: str):
    task = q.enqueue(count_words,
                     job_timeout='2h',
                     result_ttl=1000,
                     args=(url,))
    return JSONResponse(content=success_return({
        "length_queue": len(q),
        "task_id": task.id
    }))

Here, the count_words just sleeps for 10 minutes and return the content of the url

score 0 · Answer 1 · answered Jul 13 '23 at 06:08

What you are asking is similar to how OS schedules things. It can be implemented but I don't think there is a good package for accomplishing the same in python. If you want to stick with RQ and if you are fine to wait until 10 sec job get completes once before picking up all the 5 sec jobs next, you can follow one of the approaches:

Use separate queues for both the jobs and listen on two different workers.
Use a single worker with two queues and specify their priority order. You can read more about dequeuing strategies here.

It's difficult to do preemption of jobs since you have to store the state across machines and you might lose the distributed processing nature of RQ.

How to pause and resume RQ task

1 Answers1