4

For a PBS script called with qsub, I want to know how many total CPU's have actually been allocated in case the number defined in the PBS file is overwritten by inputs from the command line. For example with the following pbs script file:

jobscript.pbs:

#!/bin/bash
#PBS -N test_run
#PBS -l nodes=32
#PBS -l walltime=06:00:00
#PBS -j oe
#PBS -q normal
#PBS -o output.txt

cd $PBS_O_WORKDIR

module load gcc-openmpi-1.2.7
time mpiexec visct

This script could be run with just 16 CPU's (instead of 32) using the following command line:

$ qsub -l nodes=2:ppn=8 jobscript.pbs

So I would like a robust method for determining how many CPU's are actually available from within the script.

MasterHD
  • 2,264
  • 1
  • 32
  • 41

2 Answers2

7

I was able to answer my own question with the following solution using the $PBS_NODEFILE environment variable which contains the path to a file listing information about the available nodes:

jobscript.pbs:

#!/bin/bash
#PBS -N test_run
#PBS -l nodes=32
#PBS -l walltime=06:00:00
#PBS -j oe
#PBS -q normal
#PBS -o output.txt

# This finds out the number of nodes we have
NP=$(wc -l $PBS_NODEFILE | awk '{print $1}')
echo "Total CPU count = $NP"

Thanks to "Source" after much online searching.

MasterHD
  • 2,264
  • 1
  • 32
  • 41
  • 1
    Go ahead and accept your own answer so this question will be closed. – Wesley Bland Jul 23 '13 at 14:24
  • Is this for one or multiple jobs? Does "CPUs have actually been allocated" mean that the job is running? – Anna Jan 30 '20 at 19:45
  • @Anna This is for a single job (but you can run this job multiple times) and the script will execute once for each individual job as they start running. The result provides the number of CPU's available in the current single job. – MasterHD Jan 31 '20 at 12:44
  • @MasterHD, so does it mean that $PBS_NODEFILE contains information about a single job? What if I run a couple of jobs at the same time? – Anna Jan 31 '20 at 17:08
  • 1
    @Anna each job has its own process (private environment variables and output stream), so yes $PBS_NODEFILE will contain the number of nodes of the corresponding job. – MasterHD Feb 03 '20 at 10:35
5

MasterHD I know you have found your answer but I thought I would share another way

This code is longer but it helps for my specific needs. I actually use pbsnodes commands. Below is a snippet of my code.

@nodes_whole =`pbsnodes -av -s $server | grep "pcpus" `;
$nodes_count = ` pbsnodes -av -s $server | grep "pcpus" | wc -l `;
while($i < $nodes_count){
    @cpu_present = split(/\s+/, $nodes_whole[$i]);
    $cpu_whole_count += $cpu_present[3];
    $i++;
}

I do this because in my script I check things like the amount of cpus , which varies depending on the node the cpus maybe be 4, 8, 16. Also I have multiple clusters which are always changing size and I don't want the script have specific cluster or node info hard coded. Mainly, I do this because when a user submits a job I check to see how many resources they can use . If say they want use a queue and request 200 cpus but on cluster A their job will be queued my script can tell them they will be queued but would not be on cluster b or d. So then they have the option to change before they submit.

I also use it to check for nodes down:

@nodes_down=`pbsnodes -l -s $server `;

I see what resources are in use:

@nodes_used=`pbsnodes -av -s $server | grep "resources_assigned.ncpus" `;

Also in one case I have two clusters running off one head node while I wait for hardware. In that case I check to see what cluster the node is assigned to and then do a count based on the node assigned to that cluster. That way all the users see is another cluster and use the script they way they would for any of the other clusters.

I just mention because I have found a lot of useful ways to use the pbsnodes and it worked well for my particular needs.

Carole
  • 117
  • 9