How can I loop through specific elements of a list in Bash?

Question

I have a list of files stored in a variable obtained by entering

files="./*.fasta"

I would like to create a for loop that will loop through: the first 200 elements, elements 201-400, elements 401-578, for example.

How can I achieve this? I tried something like

for file in $files[1-200]; do 
    echo $file
done

but clearly this does not work.

is there something that differentiate those elements from the other ones??? — eduardogr, Apr 05 '18 at 12:56
No, the split is entirely arbitrary. I just want to split the files up so I can run multiple batches simultaneously. — chrispbio, Apr 05 '18 at 14:23

Inian · Accepted Answer · 2018-04-08T16:45:29.723

4

Using a variable to populate a list of files is not recommended. The best way to do it would be using arrays!

You need to enable a shell option to avoid null glob expansion by doing shopt -s nullglob so that if no files are found the for-loop exits gracefully. The example below shows iterating over the 200 files at a time. You could change the indices as needed to print from 200-400 and 400-600 as needed in the for-loop.

shopt -s nullglob
files=(*.fasta)

if (( "${#files}" >= 200 )); then
    for ((i=0; i<200; i++)); do
        printf '%s\n' "${files[i]}"
    done
fi

edited Apr 08 '18 at 16:45

answered Apr 05 '18 at 13:04

Inian

80,270
14
142
161

Thanks for the array advice. Should I understand the null glob bit as avoiding the usual error I see where no matching file is found and bash attempts to run whatever program I'm using with just "/*.fasta" as the input? – chrispbio Apr 13 '18 at 13:03
1

@chrispbio : Exactly yes! – Inian Apr 13 '18 at 13:10

score 1 · Answer 2 · answered Apr 05 '18 at 14:30

1

Put them in an array, then use substring expansion to get batches of files.

files=(./*.fasta)
for ((i=0; i< ${#fasta[*]}; i+=200)); do
    process "${files[@]:i:200}" &
done

answered Apr 05 '18 at 14:30

chepner

497,756
71
530
681

Mateusz Piotrowski · Answer 3 · 2018-04-05T15:20:32.923

The problem may be approach differently. Instead of using a for loop you may use find and xargs:

find * -name '*.fasta' -maxdepth 0 -print0 | xargs -0 -n 200 -P 0 echo

find passes every file name to xargs which in turn spawns a process (-P 0) for every 200 input files (-n 200).

This one-liner uses -print0 and -0 flags just in case your filenames contain whitespace.

The for loop construct is less than ideal in this scenario.

Alternatively, you might use a while loop and a readarray builtin:

find * -name '*.fasta' -maxdepth 0 | while readarray -n 3 a && [[ ${#a} -ne 0 ]]
do
    echo ${a[@]}
done

How can I loop through specific elements of a list in Bash?

3 Answers3