I wan to submit a sequential job, but I got:
sbatch: error: Memory specification can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available
This is my .sh file:
#SBATCH --nodes=1
#SBATCH --time=01:00:00
#SBATCH --job-name=job-8-0
#SBATCH --mem=64000mb
#SBATCH --exclusive
module purge
module load gcc-8.3.0-gcc-4.8.5-tu6ftrf
echo "Starting job-8-0"
echo "Starting at `date`"
cd code
srun gcc -Wno-return-type file1.cpp file2.cpp file3.cpp file4.cpp file5.cpp main.cpp -o myExperiment -lstdc++ -lm
srun ./myExperiment 8 0
echo "Experiment 8-0 finished with exit code $? at: `date`"
The node login01 info is:
NodeName=login01 Arch=x86_64 CoresPerSocket=8
CPUAlloc=0 CPUErr=0 CPUTot=32 CPULoad=29.91
AvailableFeatures=(null)
ActiveFeatures=(null)
Gres=gpu:8
NodeAddr=10.0.50.0 NodeHostName=login01 Version=17.11
OS=Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018
RealMemory=1 AllocMem=0 FreeMem=65001 Sockets=2 Boards=1
State=IDLE+DRAIN ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
BootTime=2021-05-25T13:13:10 SlurmdStartTime=2021-05-25T16:35:31
CfgTRES=cpu=32,mem=1M,billing=32
AllocTRES=
CapWatts=n/a
CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
Reason=gres/gpu count too low (0 < 8) [slurm@2021-06-28T13:38:40]
There are also other nodes with FreeMem=122000 , or 121000,... etc that is more than 64000mb
This is the specifications of the supercomputer:
• OS: Linux CentOS 7
• 300 compute nodes
• Each node has:
o 2 CPUs: Xeon E5-2650 8 Cores 2.000GHz (total 16 cores)
o 2 dual AMD FirePro S10000 GPUs
o Memory : 128 GB RAM
• Scheduler: Slurm
When I open nodes.conf
NodeName=login01 NodeAddr=10.0.50.0 CPUs=32 Procs=32 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=IDLE
which is the same for all nodes.
This is slurm.conf
ClusterName=sanam
ControlMachine=mgmt01
ControlAddr=10.0.1.254
SlurmUser=slurm
SlurmdUser=root
SlurmctldPort=6817
SlurmdPort=6818
AuthType=auth/munge
StateSaveLocation=/var/spool/slurm/ctld
SlurmdSpoolDir=/var/spool/slurmd
SwitchType=switch/none
MpiDefault=none
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
ProctrackType=proctrack/pgid
ReturnToService=2
GresTypes=gpu
# TIMERS
SlurmctldTimeout=300
SlurmdTimeout=300
InactiveLimit=0
MinJobAge=300
KillWait=30
Waittime=0
SelectType=select/cons_res
SelectTypeParameters=CR_Core
# SCHEDULING
SchedulerType=sched/backfill
FastSchedule=1
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/slurmdbd
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/linux
SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurm/slurmd.log
JobCompType=jobcomp/none
AccountingStorageHost=10.0.1.254
include /etc/slurm/nodes.conf
include /etc/slurm/partitions.conf
#include /etc/slurm/gres.conf
What causes this problem "Memory specification can not be satisfied"?
Should I specify the RAM and CPU with srun ? specifically the second srun that run the experiment?
