2

On our cluster, when I submit jobs requesting more than (including) 40 nodes or 640 cores, the $LSB_HOSTS gets empty and so the job stops. I use this variable to generate a nodelist file which I use with the mpirun command line as the following:

#BSUB -q cpu
#BSUB -J gromacs
#BSUB -o job.out
#BSUB -e job.err
#BSUB -n 640

#####################################################
#####################################################

INPUT=test184000atoms_verlet.tpr

echo ""
echo "----------------------- INTIALIZATIONS -----------------------------"
echo ""

source /lustre/utility/intel/composer_xe_2013.3.163/bin/compilervars.sh intel64
source /lustre/utility/intel/mkl/bin/intel64/mklvars_intel64.sh
source /lustre/utility/intel/impi/4.1.1.036/bin64/mpivars.sh

MPIRUN=/path/to/intel/impi/4.1.1.036/intel64/bin/mpirun

EXE=mdrun_mpi

if test ! -x `which $EXE` ; then
    echo
    echo "ERROR: `which $EXE` not existent or not executable"
    echo "Aborting"
    exit 1
fi

CURDIR=$PWD
cd $CURDIR
rm -f nodelist >& /dev/null
touch nodelist

for host in `echo $LSB_HOSTS`
do
echo $host >> nodelist
sleep 2
done

NP=`cat nodelist |wc -l`
NN=`cat nodelist |sort |uniq|tee nodes |wc -l`

echo
echo "Executable : `which $EXE`"
echo "Working directory is $CURDIR"
echo "Running on host `hostname`"
echo "Directory is `pwd`"
echo "This jobs runs on $NN nodes"
echo "This job has allocated $NP core(s)"
echo

ulimit -aH
echo
ls -al

echo ""
echo "----------------------- RUN -----------------------------"
echo ""

date '+RUN STARTED ON %m/%d/%y AT %H:%M:%S'

$MPIRUN -np $NN -machinefile nodes $EXE -v -deffnm  $INPUT >& $EXE.log

date '+RUN ENDED ON %m/%d/%y AT %H:%M:%S'

echo ""
echo "----------------------- DONE ----------------------------"
echo ""

ls -al

Any hints here?

Can you see something wrong with this script?

Thanks,

Éric.

Éric
  • 419
  • 5
  • 17
  • Stackoverflow isn't really for this sorts of questions. Are you absolutely sure `LSB_HOSTS` is empty? That would most likely indicate a problem with the batch system. How long are the hostnames? – Anycorn Sep 12 '13 at 05:03
  • I'm sure because when I echo $LSB_HOSTS nothing is displayed and nodelist file size is zero. hostnames start from node001 to node332. – Éric Sep 12 '13 at 05:49
  • Is there a size or length limit for bash environment variables? – Éric Sep 12 '13 at 06:01
  • alright, I found that when LSB_HOSTS exceed 4kB echo returns an empty strings. This size corresponds to 512 of our nodes, each node name having 7 bytes + 1 space. I'm now wondering how to print large environment variable. – Éric Sep 12 '13 at 14:23
  • With debug log enabled, are there any errors in the sbd log file? The shell should be OK. This simple test works for me. `echo -n "export LSB_HOSTS=\"" > t.sh; seq -f "%03g" 1 640 | while read n; do echo -n "host$n "; done >> t.sh; echo "\"" >> t.sh ; . ./t.sh ; echo $LSB_HOSTS` – Michael Closson Sep 12 '13 at 15:14
  • Ha! I knew it had to do with length. don't echo the list, try `for host in $LSB_HOSTS; do ....` – Anycorn Sep 12 '13 at 15:33
  • Still seems ok to me. `for host in $LSB_HOSTS; do echo $host >> hostlist; done`. `wc -l hostlist` shows 640 lines and `wc -c hostlist` shows 5120 chars. Using `for host in \`echo $LSB_HOSTS\` ` also works. – Michael Closson Sep 12 '13 at 16:08
  • @MichaelClosson he may have a different shell or shell setup. Maybe some twisted AIX system. – Anycorn Sep 12 '13 at 17:49
  • I think the problem is more in our LSF than the shell. Thanks to Micheal I could generate big environment variable from my login shell. Actually, I tried to do `for host in $LSB_HOSTS` and also to use `printf` but noting change. To me, if the require data exceed 4kB `LSB_HOSTS` is simply empty. I've got to understand how this variable is filled by LSF. Thanks for your help guys. – Éric Sep 13 '13 at 20:12

1 Answers1

1

I finally got a solution.

The point is that for some reason the variable LSB_HOSTS is sometime not set. Fortunately, there is another one: LSB_MCPU_HOSTS

For people who are interested, here is how I use it:

CURDIR=$PWD
cd $CURDIR
rm -f nodelist nodes n >& /dev/null
touch nodelist
touch nodes
NP=0

for host in `echo $LSB_MCPU_HOSTS | sed -e 's/ /:/g'| sed 's/:n/\nn/g'`
do
echo $host >> nodelist
echo $host | cut -d ":" -f1 >> nodes
nn=`echo $host | cut -d ":" -f2`
NP=`echo $NP+$nn | bc`
done

NN=`cat nodelist | wc -l`

echo
echo "Executable : `which $EXE`"
echo "Working directory is $CURDIR"
echo "Running on host `hostname`"
echo "Directory is `pwd`"
echo "This jobs runs on $NN nodes"
echo "This job has allocated $NP core(s)"
echo

Thank you for your help.

Éric.

Éric
  • 419
  • 5
  • 17