2

I'm trying to pass the index of a job in a job array as a parameter to another bash script.

numSims=3 
numTreatments=6 # uses numTreatments top rows of parameters.csv
maxFail=10
j=1
while [ $j -le $numSims ];
do
    bsub -q someQueue -J "mySim[1-$numTreatments]%2" ./another_script.sh $LSB_JOBINDEX $j $maxFail
    let j=j+1
done

The ultimate idea here is to submit, for each of 1,...,numTreatments,numSims jobs (simulations). I'd like two jobs running at a time (%2). Outputs have the form XX_indexNumber_simNumber, where indexNumber runs from 1,...,numTreatments and simNumber from 1,...,numSims.

Ideally, everything submitted as part of this script would have the same job ID. This isn't yet set up correctly, because all jobs with the same j are being assigned a distinct job ID. My immediate problem is that another_script.sh is not recognizing $LSB_JOBINDEX as input--it sees $j and $maxFail as the first and only two passed parameters. When I put some other variable in place of $LSB_JOBINDEX, there's no problem. What am I doing wrong?

Edit - some things I've tried: "$LSB_JOBINDEX", ${LSB_JOBINDEX}, %I, and I=$LSB_JOBINDEX; bsub ... $I $j $maxFail

Sarah
  • 1,614
  • 1
  • 23
  • 37
  • I don't see where `$LSB_JOBINDEX` is set. Since it's empty and unquoted, it's as if it's not there at all which is consistent with the behavior you report. – Dennis Williamson Jun 26 '12 at 18:22
  • From what I understand, `LSB_JOBINDEX` is automatically defined for job arrays, e.g., [here](http://rc.fas.harvard.edu/kb/high-performance-computing/lsf-submitting-lots-of-short-jobs-job-arrays/) and [here](http://www.csc.fi/english/pages/murska_guide/batch_jobs/serial_batch_jobs) – Sarah Jun 26 '12 at 18:25
  • Well, then for some reason it's not. In any case, I recommend putting quotes around it and the other variables. – Dennis Williamson Jun 26 '12 at 18:43
  • Updated the question with a few of the alternatives I've attempted. – Sarah Jun 26 '12 at 18:55
  • Try `x$LSB_JOBINDEX` and see what the script receives. If that is equal to `x`, then `$LSB_JOBINDEX` was empty (which it probably is, looking at your results). – Reinier Torenbeek Jun 26 '12 at 19:07
  • You're right, it's empty. I'm severely confused how I'm supposed to get it full. – Sarah Jun 26 '12 at 19:10
  • 2
    I have no experience with `bsub`, but from the links you sent, it looks like `$LSB_JOBINDEX` is communicated via the environment and is accessible from *within the new job*. Did you try accessing `$LSB_JOBINDEX` from inside `another_script.sh`, without passing it as a parameter? – Reinier Torenbeek Jun 26 '12 at 19:15
  • BOOM! You're right. I had missed this 'subtlety.' If you want to answer formally, I'll pick it. (Syntactically, though, I don't see how my command differs from the one in the first link.) – Sarah Jun 26 '12 at 19:29
  • OK, thanks, did it. Looking at the two links, the second one is a better read to me. The first one is confusing and even has a typo in the name of the variable. – Reinier Torenbeek Jun 26 '12 at 20:05

1 Answers1

7

From this link:

The definition above will launch not just one batch job, but 100 batch jobs where the subjob specific environment variable $LSB_JOBINDEX gets values form 1 to 100. This variable can then be utilized in the actual job launching commands so that each subtask gets processed.

In your case, this means that the variable $LSB_JOBINDEX is available from inside the script another_script.sh. You do not need to pass it as a parameter, but just access $LSB_JOBINDEX in your script.

Reinier Torenbeek
  • 16,669
  • 7
  • 46
  • 69
  • Thanks again. After looking at half a dozen simplistic uses of `$LSB_JOBINDEX`, I didn't give the good descriptions a fair chance. This lesson should percolate up to the designers of the first link... – Sarah Jun 26 '12 at 20:22
  • It is also possible to expand `$LSB_JOBINDEX` in the command itself, but I didn't manage to do it in the command arguments unfortunately: https://stackoverflow.com/questions/55472377/how-to-use-lsb-jobindex-in-bsub-array-job-arguments-in-platform-lsf – Ciro Santilli OurBigBook.com Apr 02 '19 at 10:20