task queue manager for linux

Question

I'm looking for a lightweight task queue manager for linux. The way I see it, different processes should be submitting task to the queue, the queue manager would execute them one by one in order of submission.

Is there any software that does that. I've looked at celery and geaman, but they are too heavy for the task I'm at.

Did you find a proper solution for your use case? – mdo Jul 04 '13 at 11:32 — mdo, Jul 04 '13 at 11:32
Hello @facha. Did you end up finding a suitable solution? – Shadi Jun 04 '17 at 15:04 — Shadi, Jun 04 '17 at 15:04
Finally I've used a database table as queue – facha Jun 09 '17 at 07:09 — facha, Jun 09 '17 at 07:09

score 2 · Answer 1 · answered Mar 03 '19 at 02:35

Except for task spooler, I have literally not found anything else that serves this need. Tsp is great, but I am not always in a position to compile or install on every server I have to work on.

I've been struggling with this for a while, and after 3 iterations (starting from more complex to less, actually), I've come up with a pure bash implementation that appears to work fine so far. You can find it at https://github.com/sitaramc/bq.

Bq uses the fact that on Unix, "mv" (within the same file system) is an atomic operation, for all its locking needs.

It's just one bash script so installation is trivial.

The script is liberally commented and you should be able to review it in a few minutes if you wish to.

score 1 · Answer 2 · answered Jun 28 '13 at 12:37

Previously I used the Berkely lpd for this (before cups, it was the default printing system for Linux). Although the queue management is implemented as a compiled daemon, the printer interface is a simple shell script (usually a wrapper around ghostview).

It's very stable and provides tools for managing the queue(s).

score 0 · Answer 3 · answered Jun 13 '20 at 07:52

0

I currently use GNU Parallel to run batches like this, which has the advantage that I can use all the machines I have access to at once. However, I need to be able to share execution resources between different users and better reporting.

answered Jun 13 '20 at 07:52

Leo

111
3

score 0 · Answer 4 · answered Jun 28 '13 at 10:07

0

batch executes commands when system load levels permit.

Maybe "at" can solve your requirement already?

answered Jun 28 '13 at 10:07

mdo

206
2
10

I'm not sure. At could queue tasks so that the next one starts executing when the previous is finished? – facha Jun 28 '13 at 10:40
With the at implementation in Debian/Ubuntu it defaults to running one job every minute via atd (can be configured via switch -b, the batch interval). When I fire in a couple of dozen jobs via batch they run in correct order. But there's no waiting for the previous job to be finished. So with the interval you choose you would have to take into account the runtime of your jobs. – mdo Jun 28 '13 at 11:41
You can check the output of this: `for i in $(seq 1 1 3); do echo "echo $i \$(date)>> /tmp/atqueue ; sleep 120 ; echo $i 2nd \$(date)>> /tmp/atqueue "|batch; done` – mdo Jun 28 '13 at 11:43

task queue manager for linux

4 Answers4