How to increment a variable in parallelized bash script

Question

so I have a bash script that forks new process in a double for-loop

for each date "d":
    for each instance "i":
        do-something "d" "i" &
    done
done

And in do-something, I would like to increment a counter variable. However, since it's a counter variable, it seems like accumulating doesn't work. What's the best way to solve this?

What's the fun of incrementing it in a child shell (&), when you can't get the value back to parent shell. Increment the counter in just before calling the do-something function/script etc and maintain the count. — AKS, Aug 07 '15 at 19:56
Just do `((i++))` in the line above `do-something` and each `do-something` will get its own updated copy - independent of the others. — Mark Setchell, Aug 07 '15 at 22:00

score 0 · Accepted Answer · answered Aug 07 '15 at 19:59

You can do it this way. What's the main reason of incrementing the counter in a child shell/process when you cant get the value back. Incrementing before calling do-something and then maintain your counter.

#!/bin/bash

c=0
do-something ()
{
 echo 'I do some shenzi'
 sleep 20;
}

while : ; do
    while : ; do
        echo "-- Counter == $((++c))" && sleep 5;
        do-something "d" "i"  &
    done
done

Mark Setchell · Answer 2 · 2015-08-08T11:18:31.857

You cannot pass variables from forked/backgrounded scripts to their parents or siblings, so basically you need to use a different method. Here I store your counter in the filesystem.

I chose to use Perl because I want to use its flock() function which allows me to lock the file containing the counter for exclusive access by a single process at a time - that way you avoid race conditions between the multiple, parallel processes.

So, save the following little Perl script as BumpCounter

#!/usr/bin/perl
use strict;
use warnings;
use Fcntl qw(:seek :flock);

# Open counter file /tmp/counter.txt
open my $fh, '+<', '/tmp/counter.txt' or die "Couldn't open file: $!";

# Lock it for exclusive access
flock($fh,LOCK_EX) or die "couldn't get lock: $!\n";

# Read current value
my $cnt=<$fh>;

# Increment value and write back
seek($fh,0,SEEK_SET);
printf $fh ++$cnt;

# Close handle
close $fh;

Then make it executable by running the following command:

chmod +x BumpCounter

Then inside your do-something script you would do

# do some work

./BumpCounter

# do some work

To see the counter's value, just do

cat /tmp/counter.txt

To test it, zero the counter, start 500 do-somethings in the background, then check the counter like this:

echo 0 > /tmp/counter.txt
for j in {0..499}; do 
   ./do-something &
done
cat /tmp/counter.txt

500

If you like this approach, the script could be adapted to accept a parameter, so that it could work like this:

./Counter RESET     # reset counter to zero
./Counter READ      # read counter's current value
./Counter +3        # add 3 to counter
./Counter -1        # subtract 1 from counter

How to increment a variable in parallelized bash script

2 Answers2