0

I'm trying to figure out why I'm seeing diminishing speed returns when backgrounding lots of processes in a Bash script. Something like:

function lolecho() {
    echo "lol" &> /dev/null
}
c=0
while true; do
    for i in $(seq 1 1000); do
        lolecho &
        ((c+=1))
        if [[ $((c%1000)) -eq 0 ]]; then
            echo "Count $c"
        fi
    done
    sleep .1
done

It screams out of the gate up to 10,000, 20,0000... but it then starts to slow down in how quickly it can put up backgrounded processes around 70,000... 80,0000. As in, the rate at which the count prints to screen slows down by a seemingly linear amount, depending on the total.

Should not the rate at which the machine can run background jobs that finish basically instantly be consistent, regardless of how many have been added and closed?

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Locane
  • 2,886
  • 2
  • 24
  • 35
  • (1) "Instantly" may not be all that instant. If it's faster to do a fork than to do an exec and let the dynamic linker / loader / &c. start up your target process, one can fall behind. (2) Eventually, you're going to be requiring the process table to be cleaned up before new PIDs can be found. – Charles Duffy Dec 03 '20 at 22:19
  • 1
    What I'd strongly suggest doing here, instead of asking for someone to give you an answer based on theory, is to pull out a full-system tracing toolkit like [sysdig](https://github.com/draios/sysdig) and look at what's _actually happening in practice_. "Tell me what my process table looks like 50ms into my trace", or "tell me which syscalls have more than X amount of latency" are valuable tools. – Charles Duffy Dec 03 '20 at 22:20
  • 1
    Are you sure that the number of processes concurrently running approaches a maximum over time? If the time it takes to "start starting one" (i.e., the time the parent spends processing the line `lolecho &` -- by that time the child process may not be completely initialized!), is smaller than the time the child process spends from initialization to finalization; then the number of processes will not have an upper bound. In fact, I think there are only two possibilities: At most a single child process at any given time; or else an infinitely growing number of them. – Peter - Reinstate Monica Dec 03 '20 at 22:21
  • Also, the more running processes you have at any time, the longer it takes for any process to get time on a CPU core scheduled, so the higher your chances of falling behind are (wrt. the newly forked processes not having time to get linked and loaded, and then do their very quick work and exit before another process is created behind them). – Charles Duffy Dec 03 '20 at 22:22
  • @CharlesDuffy Right, that's a feedback loop. CPU polar cap ice melt. – Peter - Reinstate Monica Dec 03 '20 at 22:23
  • 2
    Anyhow, this is a "measure actual behavior with your real workload" question, not a "ask people on Stack Overflow who can't inspect your actual system" question. – Charles Duffy Dec 03 '20 at 22:24

2 Answers2

0

The answer was to use the Linux built-in wait command:

function lolecho() {
    echo "lol" &> /dev/null
}
c=0
while true; do
    for i in $(seq 1 1000); do
        lolecho &
        ((c+=1))
        if [[ $((c%1000)) -eq 0 ]]; then
            echo "Count $c"
        fi
    done
    wait   # <------------
done

The script now produces processes consistently and faster in general.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Locane
  • 2,886
  • 2
  • 24
  • 35
  • 1
    What you're doing here is stopping every so often and letting the background processes finish so they never get so numerous as to overwhelm other content in the process table. I don't think it actually answers your question about diminishing returns. – Charles Duffy Dec 03 '20 at 23:38
  • 1
    I'd also argue that you don't need the `sleep .1` as long as you're `wait`ing. – Charles Duffy Dec 03 '20 at 23:39
0

A bit long for a comment ... OP's solution of using the wait command is fine but could probably be fine tuned a bit ...

As coded (in OPs answer):

  • 1K background processes are spawned (likely hitting some contention on system resources)
  • we wait for all 1K processes to finish before we ...
  • start a new set of 1K processes

For a more consistent throughput I'd want to:

  • look at limiting the number of concurrent background processes (eg, 50? 100?); will need to run some tests on your particular system) to reduce system resource contention then ...
  • use wait -n to start up a new process as soon as one finishes

Granted, this may not make much difference for this simple example (lolecho()) but if doing some actual work you should find you maintain a fairly steady workload.

A couple examples of using wait -n: here and here - see 2nd half of answer

If using an older version of bash that does not support the -n flag, an example using a polling process

markp-fuso
  • 28,790
  • 4
  • 16
  • 36