bash - for loop through multiple directories and their files

Question

I have 5 directories in the same path my_path/:

my_path/1951 
my_path/1952 
my_path/1953 
my_path/1954 
my_path/1955

Each directory contains a different number of netcdf files ending in .nc.

And I need to perform a CDO command by looping in each directory and files. The command below applies only to the first directory and files my_path/1951/*.nc:

for i in my_path/1951/*.nc
do
   cdo selyear,1951/1970 "$i" "$i"2
done

The command select years from the nc files in the directory starting from the year of the directory and ending 20 years later.

So for the second directory it will be:

for i in my_path/1952/*.nc
do
   cdo selyear,1952/1971 "$i" "$i"2
done

and for the third:

for i in my_path/1953/*.nc
do
   cdo selyear,1953/1972 "$i" "$i"2
done

etc...

Is there a way I can do everything with a unique for loop or nested for loop instead of repeating the for loop above for each directory?

tshiono · Accepted Answer · 2022-03-10T10:31:16.253

2

Would you please try the following:

#!/bin/bash

for i in my_path/*/; do
    year=${i%/}; year=${year##*/}       # extract year
    year2=$(( year + 19 ))              # add 19
    for j in "$i"*.nc; do
        echo cdo "selyear,${year}/${year2}" "$j" "$j"2
    done
done

It outputs command lines as a dry run. If it looks good, drop echo and run.

edited Mar 10 '22 at 10:31

answered Mar 10 '22 at 10:21

tshiono

21,248
2
14
22

1

wow a bash for loop looking estetically nice – aaaaa Mar 10 '22 at 10:26
2

@aaaaa: Opinion-based. I much prefer the `do` on a line of its own, like you did in your question. And having multiple statements in one line is generally frowned upon, regardless the language. ;-) – DevSolar Mar 10 '22 at 10:31
definitely. de gustibus non est disputandum :P – aaaaa Mar 10 '22 at 10:38
1

@DevSolar thank you for the comment. I may be affected by K&R C coding style which puts the left curly brace in the same line. The benefit is I can see more lines in the screen IMHO. As for the assignments of `year`, I just wanted to save the comment. Apart from them, I do agree with your opinion. – tshiono Mar 10 '22 at 10:48
1

@tshiono: I came a bit late for K&R and went right for C++ (and Stroustrup's ANSI style). ;-) – DevSolar Mar 10 '22 at 11:52

score 1 · Answer 2 · answered Mar 10 '22 at 11:07

1

You could loop like this:

shopt -s nullglob
for i in my_path/195[1-5]/*.nc
do
  ...
done

The shopt command ensures that the loop works even if some of the directories don't contain a nc file.

answered Mar 10 '22 at 11:07

user1934428

19,864
7
42
87

thanks. but what is inside the loop changes too with the given directory, or is not the same command for all directories. – aaaaa Mar 10 '22 at 12:40
I know, but you are calculating now already (i.e. in the attempt you have posted) `year` and `year2`. Since you oviously know how to take apart a variable, I assume that you can do a similar calculation for the new loop. – user1934428 Mar 10 '22 at 12:53

score -2 · Answer 3 · edited Mar 10 '22 at 10:28

-2

for x in $@
 for i in my_path/$x/*.nc
  ...

you can do On2 for with getting parameters of sh script.

yourScript.sh 1951 1952 ...

After that you can pass all folder to your script.

yourScript.sh ./*

edited Mar 10 '22 at 10:28

DevSolar

67,862
21
134
209

answered Mar 10 '22 at 10:25

Burak KILINC

1
2

This does not address the fact that the parameter after the comma has to change for each directory. – chutz Mar 10 '22 at 17:42

bash - for loop through multiple directories and their files

3 Answers3