Using Flock in Bash so Request is Made Only Once

Question

I'm trying to configure my script in such a way that

If some data isn't available, try to fetch it
If another process is already fetching it, wait for that process to finish
Use the data

And from here I found this very nice example of flock:

exec 200>$pidfile
flock -n 200 || exit 1
pid=$$
echo $pid 1>&200

And this fails if it can't aquire the lock (-n flag).

Can I assume that this means another file has locked the $pidfile, and how can I detect that the lock has been released in a different process?

I understand that wait $pid would wait until that process is complete, and so if there's some way to record which process currently holds the lock or just detect the unlocking so that other processes know once the data is available, then I think this will work.

Any ideas?

score 0 · Answer 1 · answered Jun 27 '18 at 09:46

0

As per the flock (1) man page,

if the lock cannot be immediately acquired, [in the absence of a -w timeout], flock waits until the lock is available

You can use fuser to see which process is holding a file handle.

answered Jun 27 '18 at 09:46

tripleee

175,061
34
275
318

I've added my current attempt so far that I think achieves the functionality I need, but right now every process fails to lock the file. – Jordan Mackie Jun 27 '18 at 13:39
1

I haven't examined your code in detail, but having two lock files seems like a disastrous recipe for deadlock, especially since you only use the *inner* lock from the `then` branch in the `else` branch. Just lock one thing, and make sure none of your code touches any of the critical resources wthout obtaining that lock. – tripleee Jun 27 '18 at 15:10
Realised that, trying locking only the one resource now. – Jordan Mackie Jun 27 '18 at 15:35

score 0 · Accepted Answer · answered Jul 05 '18 at 08:27

My solution uses two files, pid.temp and data.temp:

backgroundGetData() {
    local data=$1

    # if global is null, check file.
    if [ -z "$data" ]; then
        data=$( cat $DATA_TEMP_FILE 2>/dev/null )
    fi

    # if file is empty, check process is making the request
    if [ -z "$data" ]; then
        for i in {1..5}; do
            echo "INFO - Process: $BASHPID - Attempting to lock data temp file" >&2
            local request_pid=$( cat $PID_FILE 2>/dev/null )
            if [ -z "$request_pid" ]; then request_pid=0; fi
            local exit_code=1
            if [ "$request_pid" -eq 0 ]; then
                ( flock -n 200 || exit 1
                    echo "INFO - Process: $BASHPID - Fetching data." >&2
                    echo "$BASHPID">"$PID_FILE"
                    getData > $DATA_TEMP_FILE
                ) 200>$DATA_TEMP_FILE
                exit_code=$?
            fi

            echo "INFO - Process: $BASHPID - returned $exit_code from lock attempt.">&2
            [ $request_pid -ne 0 ] && echo "INFO - Process: $BASHPID - $request_pid is possibly locking">&2
            if [ $exit_code -ne 0 ] && [ $request_pid -ne 0 ]; then
                echo "INFO - Process: $BASHPID - waiting on $request_pid to complete">&2
                tail --pid=${request_pid} -f /dev/null
                echo "INFO - Process: $BASHPID - finished waiting.">&2
                break
            elif [ $exit_code -eq 0 ]; then break;
            else
                sleep 2
            fi
        done
        data=$( cat $DATA_TEMP_FILE )
        if [ -z "$data" ]; then
            echo "WARN - Process: $BASHPID - Failed to retrieve data.">&2
        fi
    fi
    echo "$least_loaded"
}

And it can be used like so:

DATA=""
DATA_TEMP_FILE="data.temp"
PID_FILE="pid.temp"
$( backgroundGetData $DATA ) & ## Begin making request

doThing() {
    if [ -z $DATA ]; then
        # Redirect 3 to stdout, then use sterr in backgroundGetData to 3 so that
        # logging messages can be shown and output can also be captued in variable.
        exec 3>&1
        DATA=$( backgroundGetData $DATA 2>&3)
    fi
}

for job in "$jobs"; do
    doThing &
done

It's working for me, though I'm not 100% sure on how safe it is.

Using Flock in Bash so Request is Made Only Once

2 Answers2