1

I am using Centos on a HPC to run my code. I typically have a folder that contains a run_calc File, which is what I want to run as:

qsub run_calc

Now I want to write a script "submit_all.sh" that submits all run_calc files in all the subfolders in their current directory and not from the from a parent folder where I runt the submit_all.sh script.

I found similar questions posted here Solution and here Solution2 that seems to be a partial answer to this question. I am not confident just submitting scripts until I found a solution which is why I ask:

In the second link I found this solution:

for i in {1..1000}; do
    cd "$i"
    qsub submit.sh
    cd ..
done

were "i" was a list of folders with the names 1-100. Is it somehow possible to use find to create a list of all the subdirectories and path it to the for loop? How would i deal with subsubdirectories? Would I be able to change the cd .. statement such that I always go back to the parent folder directly in that case?

I fond this solution here: Solution

#!/bin/sh

for i in `find /var/www -type d -maxdepth 1 -mindepth 1`; do
   cd $i
   # do something here
done

But I do not understand what is going on? Is it possible to change the above script to the only dive into folders containing a run_calc File and also include subsubdirectries?

Thank you in advance

Farhana Naaz Ansari
  • 7,524
  • 26
  • 65
  • 105
NorrinRadd
  • 545
  • 6
  • 21
  • Use `find` in a `for loop` with caution: https://unix.stackexchange.com/questions/321697/why-is-looping-over-finds-output-bad-practice – iamauser Feb 02 '18 at 15:08

2 Answers2

1

Assuming that you are using bash as your shell:

$ cat ./test.sh
#!/bin/bash

IFS=$'\n' 
while read -r fname ; 
do 
    pushd $(dirname "${fname}") > /dev/null
    qsub run_calc 
    popd > /dev/null 
done < <(find . -type f -name 'run_calc')

find . -type f -name 'run_calc' finds all paths to file run_calc inside the current directory and its subdirectories. This is input for the while loop.

pushd, popd are bash specific, and adds in or pops out of directory stack.

iamauser
  • 11,119
  • 5
  • 34
  • 52
  • Whenn i submit this script i get "qsuball.sh: line 1: $: command not found ". if i delete line one the script is executed, but has not result it seems. Am i doing something wrong? – NorrinRadd Feb 02 '18 at 18:49
  • What's in line#1 of your script ?Content of the script excludes this line: `$ cat ./test.sh`. I hope you are not including that. – iamauser Feb 02 '18 at 18:53
  • I deleted the cat ./test.sh part but left the $ somehow... unfortunately if i start with #!/bin/bash it still doesnt work. at least there are no new jobs in the queue if i have a look at it? Thank you for taking the time btw! – NorrinRadd Feb 02 '18 at 21:04
  • Check that this command gives you the right path for your run_calc file: `find /var/www -type f -name 'run_calc'`. If it does, then go to one of those directories and execute `qsub run_calc` and see if that actually works. – iamauser Feb 02 '18 at 21:12
  • If i run: `find /var/www -type f -name 'run_calc'` there is no error but, nothing is found. I just get a new empty command line if that makes sense. – NorrinRadd Feb 03 '18 at 16:30
  • If you don’t find any “run_calc” file then how will qsub run it ? Find looks for “run_calc” file inside “/var/www” ? – iamauser Feb 03 '18 at 23:16
  • Dear iamauser, i am really a beginner when it comes to linux systems. I am able to do easy file manipulations with vim and navigated copy etc.... Unfortunately i do not know what the “/var/www" directory is. What i want to do is to search all the sub directories contained in my current working directory and submit the 'run_calc Files'. If possible i would also want to include deeper directories (sub sub directories) . I can assure you that all the sub directories inside the working dir where i test the command contained a 'run_calc File'! I could submit these files manually! – NorrinRadd Feb 04 '18 at 12:14
  • You had “/var/www” in your question, that’s why i put that in the answer. If you want to find files under the current directory and subdirectories, then replace “/var/www” with “.” (dot). Also see my updated answer. – iamauser Feb 04 '18 at 14:28
  • Sorry, i copy and pasted from the links i provided without really understanding what i read. Thank you, it seems to work now! – NorrinRadd Feb 07 '18 at 11:16
0
for d in `find . -type d`
do ( cd "$d"
     if test ! -f run_calc; then continue; fi
     qsub run_calc
) done

( commnds ) execute commands in a separate process and effect of cd does not "leak".

slitvinov
  • 5,693
  • 20
  • 31