How can I queue multiple jobs, but limit it to run 5 at a time, using Bash

Question

I have an text file that holds a list of jobs to process. It could process them all at once, but it overloads computer resources and some jobs fail. It could also process one line at a time, but the overall processing time is too long.

My customer is asking us to process 5 at a time to see if we can speed up processing time without missing jobs along the way. I feel like this is possible and likely pretty easy, but my brain is stuck on it and I can't figure it out.

Each job is on one row of the text file. Is there a way to load a list of jobs to run, but limit it to running 5 at a time?

score 1 · Answer 1 · answered Nov 03 '17 at 19:21

1

this should execute in the manner you are looking for. I have not tested it though. It will work only on BASH.

IFS=$'\n' read -d '' -r -a jobs < list_of_jobs.txt
total_number_of_jobs=$( wc -l list_of_jobs.txt | awk '{print $1}' )
i=0;
accumulated=0;
limit=5; #How many to process at the same time
while [ $i -lt $total_number_of_jobs ]
do
    while [ $accumulated -lt $limit ] && [ $i -lt $total_number_oj_jobs ]
    do
        ./${jobs[$i]} &
        (( i ++ ))
        (( accumulated ++ ))
    done
    wait
    accumulated=0

done

Let me know if it helps you. Regards!

answered Nov 03 '17 at 19:21

Matias Barrios

4,674
3
22
49

2

`wc -l – Charles Duffy Nov 03 '17 at 20:02
1

Compare that to what `xargs -P "$limit"` will do, where you *always* have five jobs running at a time, starting another as soon as one exits. – Charles Duffy Nov 03 '17 at 20:03
In the first line what does "jobs" mean? I understand list_of_jobs.txt is the input file, but I don't understand jobs. Also, I'm researching xargs now... – Emile Nov 03 '17 at 20:11
Charles, do I put `"$5"` where you have `"$limit"`? – Emile Nov 03 '17 at 20:16
@CharlesDuffy I got your points they are all pretty valid. I did not know about the -P for xargs. nice. – Matias Barrios Nov 06 '17 at 15:14
@Emile, that code was assuming `limit=5`; if you want to hardcode it, just `xargs -P 5`. – Charles Duffy Nov 06 '17 at 15:57

score 1 · Answer 2 · answered Nov 03 '17 at 20:00

1

I would recommend GNU Parallel as follows:

parallel -j 5 -a YourJobFile

There are lots of options, try adding —-eta or —-progress

answered Nov 03 '17 at 20:00

Mark Setchell

191,897
31
273
432

How can I queue multiple jobs, but limit it to run 5 at a time, using Bash

2 Answers2