How can I make bash 'pop' and 'shift' functions that actually return the item popped or shifted?

Question

Running

$ echo $BASH_VERSION
4.3.42(1)-release

given these two functions:

ashift ()
{
    declare -n arr;arr="$1"
    ((${#arr[@]} == 0)) && return
    echo "${arr[0]"}
    arr=("${arr[@]:1}")
}
apop ()
{
    declare -n arr="$1";shift
    ((${#arr[@]} == 0)) && return
    echo "${arr[-1]}"
    arr=("${arr[@]:0:$((${#arr[@]}-1))}")
}

the 'natural' way to use them would be

declare -a thearray
thearray=(a b c d e f g)
p=$(apop thearray)
s=$(ashift thearray)
echo "p=$p, thearray=${thearray[@]}, s=$s"

However, the output is not what you would expect:

p=g, thearray=a b c d e f g, s=a

That is because (I think) we are running the ashift and apop in a subshell to capture the output. If I do not capture the output:

declare -a thearray
thearray=(a b c d e f g)
apop thearray
ashift thearray
echo "thearray=${thearray[@]}"

the output (intermixed with the commands) is

g
a
thearray=b c d e f

So, does anyone know how I can run the apop and ashift commands in the current process AND capture the output?

Note: For completeness, these work because there is no capturing, so you don't ever run them in a subshell:

aunshift ()
{
    declare -n arr;arr="$1";shift
   arr=("$@" "${arr[@]}")
}
apush ()
{
    declare -n arr;arr="$1";shift
    arr+=("$@")
}

Short answer: Not possible on bash (or any shell that runs pipeline components in a subshell environment) without an intermediate file or a named pipe — oguz ismail, Nov 25 '19 at 19:39
Ah - and by the time I set up a temp file or a named pipe AND clean up after the fact, it would probably be faster just to code the element copy and array manipulation in line. Oh well. Thank you. — mpersico, Nov 25 '19 at 19:41
And even then, it is the contents of the modified array that you would have to echo into the temp file or pipe and read back out in the current, not sub, process. So now you'd have to do two calls - one to get the element, one to adjust the array. Hmm. I **could** do ``` p=$(apop thearray);apop thearray>/dev/null ``` if I was lazy. — mpersico, Nov 25 '19 at 19:50
You could make your functions assign to a result variable instead of printing. `foo(){ : ...; eval "${2-res}=..."; }; foo ...; var="$res"; foo ... var` (cf. `$?`) — jhnc, Nov 25 '19 at 20:24
`p=$(apop thearray);apop thearray>/dev/null` note that only one back-tick is needed on each side of your code BUT for some reason, leading/trailing spaces break the formatting (with triple or single back-quotes). Interesting problem. Good luck. — shellter, Nov 25 '19 at 21:57

dash-o · Answer 1 · 2019-12-05T06:18:34.577

Assuming that you are OK with the following 'API' to the 4 operations:

apush array value1 value2 value3 ...
ashift array value1 value2 value3 ...
apop array var1 var2 var3        # Extracted values stored in variables
aunshift array var1 var2 var3    # Extract values stored in variables

Possible to use bash reference variables (declare -n).

#! /bin/bash

function apush {
    declare -n _array=$1
    shift
    _array+=("${@}")
}

function apop {
    declare -n _array=$1
    shift
    declare _n=0
    for _v ; do
        declare -n _var=$_v
        let ++_n
        _var=${_array[-_n]}
    done
    array=("${_array[@]:0:$((${#_array[@]}-_n))}")
}

Use the following test case

A=()
apush A B1 B2 B3
apush A C1 C2 C3
echo "S1=${A[*]}"
apop A X1 X2 X3
echo "X1=$X1, X2=$X2, X3=$X3, A=${A[*]}"

Output

S1=B1 B2 B3 C1 C2 C3
X1=C3, X2=C2, X3=C1, A=B1 B2 B3

Likewise, the ashift, aunshift can be implemented following similar pattern.

Also note: because of the way bash references work, it is not possible to access variables with the same name as local variables. If the function is called with a variable matching local name, the it will produce an error. Functions modified to use _ prefix for variable, to reduce chance of this problem.

apush _array ABC

Output:
bash: declare: warning: array: circular name reference
bash: warning: array: circular name reference

AH! For those functions that return values, pass in variables in which to put the values. That eliminates the return-via-echo subshell problem. Very interesting. And it looks symmetrical; they all look like: func array val [ val ...] I bet I could even rig it so that i could write ( declare -a tgt apop array -tgtarray tgt -n 3 ) and pop the values into one variable - an array. — mpersico, Dec 05 '19 at 05:19

score 0 · Accepted Answer · edited Jun 20 '20 at 09:12

Original answer

As per the first comment: Not possible on bash (or any shell that runs pipeline components in a subshell environment) without an intermediate file or a named pipe.

Given that the actual code that does something in each case is a one-liner, it's best to just learn the idiom and use it in-line. That is what I have done in all of the code I have that was trying to use these functions.

Update

As per the solution from @dash-o below, the idea is to provide variables in which to pop or shift the removed values. Here is the full set of functions:

apush ()
{
    declare -n arr="$1"
    shift
    arr+=("$@")
}

apop ()
{
    declare -n array=$1
    shift
    declare _i=0
    if [[ $1 =~ ^- ]]
    then
        while (($#))
        do
            if [[ $1 = '-a' ]]
            then
                declare -n output=$2
                shift;shift
                continue;
            fi
            if [[ $1 = '-n' ]]
            then
                declare c=$2
                shift;shift
                continue
            fi
            echo "$1 is an invalid option"
            return 1
        done

        while ((_i<c))
        do
            ((_i+=1))
            output+=("${array[-_i]}")
        done
    else
        for _v ; do
            declare -n _var=$_v
            ((_i+=1))
            # shellcheck disable=SC2034 #https://github.com/koalaman/shellcheck/wiki/SC2034
            _var=${array[-_i]}
        done
    fi
    array=("${array[@]:0:$((${#array[@]}-_i))}")
}

aunshift ()
{
    declare -n arr="$1"
    shift
    arr=("$@" "${arr[@]}")
}

    ashift ()
{
    declare -n array=$1
    shift
    declare _i=-1
    if [[ $1 =~ ^- ]]
    then
        while (($#))
        do
            if [[ $1 = '-a' ]]
            then
                declare -n output=$2
                shift;shift
                continue;
            fi
            if [[ $1 = '-n' ]]
            then
                declare c=$2
                shift;shift
                continue
            fi
            echo "$1 is an invalid option"
            return 1
        done

        while ((_i+1<c))
        do
            ((_i+=1))
            output+=("${array[_i]}")
        done
    else
        for _v ; do
            declare -n _var=$_v
            ((_i+=1))
            # shellcheck disable=SC2034 #https://github.com/koalaman/shellcheck/wiki/SC2034
            _var=${array[_i]}
        done
    fi
    array=("${array[@]:((_i+1))}")
}

I changed the _n to _i because the variable represents an index into the array, not a number or a count (and it was too visually clashing with the -n option to declare).

Also from @dash-o's answer below, here are the docs in how to use them:

apush    array value1 value2 value3 ...
ashift   array value1 value2 value3 ...

apop     array var1 var2 var3 ...   # Extracted values stored in variables.
apop     array -n 3 -a tgtarray     # 3 extracted values stored in array
                                    # 'tgtarray'. Note that the values are
                                    # ADDED to any existing values already
                                    # in the 'tgtarray'.

aunshift array var1 var2 var3 ...   # Extracted values stored in variables.
aunshift array -n 4 -a tgtarray     # 4 extracted values stored in array
                                    # 'tgtarray'. Note that the values are
                                    # ADDED to any existing values already
                                    # in the 'tgtarray'.

score 0 · Answer 3 · answered Sep 18 '22 at 15:31

You can just use unset to remove an array element, and combine with negative indexing to refer to the last element:

A=(1 2 3 4)
H=${A[0]}
T=${A[-1]}
unset A[0]
unset A[-1]
echo Head: $H - array: ${A[@]} - tail: $T

Output:

Head: 1 - array: 2 3 - tail: 4

(This won't work well with empty arrays.)

How can I make bash 'pop' and 'shift' functions that actually return the item popped or shifted?

3 Answers3

Original answer

Update