loop with more than one item at a time

Question

We can iterate over a set of items, considering one at a time, like this:

#!/bin/bash
for i in $( ls ); do
    echo item: $i
done

How can we process several items at a time in a similar loop? Something like:

#!/bin/bash
for i,j,k in $( ls ); do
    echo item i: $i
    echo item j: $j
    echo item k: $k
done

That second shell script is incorrect but should give an accurate illustration of what I am trying to achieve.

Answer 1

If filenames does not contain spaces:

find . -maxdepth 1 | xargs -L 3 | while read i j k; do
  echo item i: $i
  echo item j: $j
  echo item k: $k
done

Edit:

I removed -print0 and -0.

Answer 2

Assuming you don't have too many items (although the shell should be able to handle quite a few positional arguments.

# Save the original positional arguments, if you need them
original_pp=( "$@" )
set -- *
while (( $# > 0 )); do
    i=$1 j=$2 k=$3     # Optional; you can use $1, $2, $3 directly
    ...
    shift 3 || shift $#   # In case there are fewer than 3 arguments left
done

# Restore positional arguments, if necessary/desired
set -- "${original_pp[@]}"

For POSIX compatibility, use [ "$#" -gt 0 ] instead of the ((...)) expression. There's no easy way to save and restore all the positional parameters in a POSIX-compatible way. (Unless there is a character you can use to concatenate them unambiguously into a single string.)

Here is the subshell jm666 mentions:

(
    set -- *
    while [ "$#" -gt 0 ]; do
        i=$1 j=$2 k=$3
        ...
        shift 3 || shift $#
    done
)

Any changes to parameters you set inside the subshell will be lost once the subshell exits, but the above code is otherwise POSIX-compatible.

Answer 3

Bit late answer, I would do this with non-spectacular way :), like:

while read -r -d $'\0' f1
do
        read -r -d $'\0' f2
        read -r -d $'\0' f3

        echo "==$f1==$f2==$f3=="

done < <(find test/ ... findargs... -print0)

Answer 4

To get to get n items a time from the list I think you want to get n items from an array.

Use it like this:

n=3
arr=(a b c d e)

echo "${arr[@]:0:$n}"
a b c

Answer 5

Here is the another solution in a typical programming manner ->

#!/bin/bash
shopt -s nullglob
arr=(*) #read the files/dirs in an array
total=${#arr[@]} #get the array size
count=0;
#loop it thru in multiples of three
while [ $count -lt $((total-2)) ]
do
        echo "i is ${arr[$count]}"
        echo "j is ${arr[$((count+1))]}"
        echo "k is ${arr[$((count+2))]}"
        count=$((count+3))
done
#print the remaining element(s)
rem=$((total%3));    
if [ $rem -eq 1 ];
then
        echo "i is ${arr[$total-1]}"
elif [ $rem -eq 2 ];
then
        echo "i is ${arr[$total-2]}"
        echo "j is ${arr[$total-1]}"
fi
echo "Done"

Answer 6

If you have GNU Parallel you can run:

ls | parallel -N3 "echo item i: {1}; echo item j: {2}; echo item k: {3}"

All new computers have multiple cores, but most programs are serial in nature and will therefore not use the multiple cores. However, many tasks are extremely parallelizeable:

• Run the same program on many files
• Run the same program for every line in a file
• Run the same program for every block in a file

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

Installation

A personal installation does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Answer 7

You can use xargs, awk, sed or paste to restructure your input.

job_select()
{
  ls
}

job_process()
{
  while read i j k; do
    printf 'item i: %s\nitem j: %s\nitem k: %s\n' "$i" "$j" "$k"
  done
}

job_restructure_xargs()
{
  xargs -L 3
}

job_rstructure_awk()
{
  awk '(NR % 3 == 1) { i = $0 } (NR % 3 == 2) { j = $0 } (NR % 3 == 0){ k = $0; print(i,j,k)}'
}

job_restructure_sed()
{
  sed -e 'N;N;s/\n/ /g'
}

job_restructure_paste()
{
  paste - - -
}

Then any of the combinations

job_select | job_restructure_xargs | job_process
job_select | job_restructure_awk | job_process
job_select | job_restructure_sed | job_process
job_select | job_restructure_paste | job_process

does what you want.