2

I'm a newbie to slurm and I'm trying to configure my bash script to, in the case that a job fails, email the corresponding standard output file when notifying me. I've managed to configure email notifications, but how can I make the body of the email contain standard output? 

#!/bin/bash
#SBATCH -n 2                    # two cores
#SBATCH --mem=3G
#SBATCH --time=48:00:00         # total run time limit (HH:MM:SS)
#SBATCH --mail-user=rylansch
#SBATCH --mail-type=FAIL

export PYTHONPATH=.
python -u model_train.py        # -u flushes output buffer immediately

I don't see answers How to configure the content of slurm notification emails? or How to let SBATCH send stdout via email?

Rylan Schaeffer
  • 1,945
  • 2
  • 28
  • 50
  • The first link seems to cover your use case, which parts of it are you having trouble understanding? – tripleee Feb 06 '20 at 06:33

2 Answers2

3

See my solution here

#!/bin/bash
#SBATCH -J MyModel
#SBATCH -n 1 # Number of cores
#SBATCH -t 1-00:00 # Runtime in D-HH:MM
#SBATCH -o JOB%j.out # File to which STDOUT will be written
#SBATCH -e JOB%j.out # File to which STDERR will be written
#SBATCH --mail-type=BEGIN
#SBATCH --mail-user=my@email.com

secs_to_human(){
    echo "$(( ${1} / 3600 )):$(( (${1} / 60) % 60 )):$(( ${1} % 60 ))"
}
start=$(date +%s)
echo "$(date -d @${start} "+%Y-%m-%d %H:%M:%S"): ${SLURM_JOB_NAME} start id=${SLURM_JOB_ID}\n"

### exec task here
( << replace with your task here >> ) \
&& (cat JOB$SLURM_JOB_ID.out |mail -s "$SLURM_JOB_NAME Ended after $(secs_to_human $(($(date +%s) - ${start}))) id=$SLURM_JOB_ID" my@email.com && echo mail sended) \
|| (cat JOB$SLURM_JOB_ID.out |mail -s "$SLURM_JOB_NAME Failed after $(secs_to_human $(($(date +%s) - ${start}))) id=$SLURM_JOB_ID" my@email.com && echo mail sended && exit $?)

you can also edit this to send seperate stdout/stderr logs or attach them as files.

This snippet is also shared on github-gists

elucida
  • 96
  • 5
0

As a regular user, you do not get to choose the contents of the email sent to you. Only the administrators can do that.

But you could add a command at the end of your submission script to send you an email, like is explained here

damienfrancois
  • 52,978
  • 9
  • 96
  • 110