10

Slurm can notify the user by email when certain types of events occur using options such as --mail-type and --mail-user.

The emails I receive this way contain a void body and a title that looks like :

SLURM Job_id=9228 Name=toto Ended, Run time 07:32:31, COMPLETED, ExitCode 0

I'd like to configure slurm so that the title or even better the body of the email contains other informations in a similar way of what the slurm command squeue --format returns.

(actually I'd like the email to contain the comment I set up using sbatch --comment)

Johann Bzh
  • 834
  • 3
  • 10
  • 25

4 Answers4

2

To customise the email sent by Slurm, you typically write a script and set the value of MailProg to the path to that script in your slurm.conf.

From the doc, MailProg is:

Fully qualified pathname to the program used to send email per user request. The default value is "/bin/mail" (or "/usr/bin/mail" if "/bin/mail" does not exist but "/usr/bin/mail" does exist).

The contrib directory of the Slurm source contains a script written in Perl that you can use and customise: https://github.com/SchedMD/slurm/tree/master/contribs/seff

damienfrancois
  • 52,978
  • 9
  • 96
  • 110
  • 1
    I am not an administrator. How do I display the contents of the `slurm.conf` file? – Charlie Parker Apr 01 '20 at 16:26
  • 1
    in your answer, you say `you typically write a script`. What is the purpose of that script? Do you mind providing a small example of such a script that modifies the body of the e-mail to be sent? If you have one in Python that'd be ideal! :-) – Charlie Parker Apr 01 '20 at 16:33
  • 1
    Is `/usr/bin/mail` a directory or a file? What is that suppose to be? – Charlie Parker Apr 01 '20 at 17:51
  • 1
    If you are not the administrator, you will not be able to modify the Slurm configuration. The script I am talking about can be the `smail` script in the referred-to GitHub repository. `/usr/bin/mail` is a program used to send emails on the Linux command line – damienfrancois Apr 01 '20 at 18:49
  • 1
    so I cannot as a user choose the body of the e-mail sent to me? – Charlie Parker Apr 01 '20 at 22:14
  • Not the one sent automatically. But you still add a command in your submission script to send an email. – damienfrancois Apr 02 '20 at 10:14
2

See my solution here

#!/bin/bash
#SBATCH -J MyModel
#SBATCH -n 1 # Number of cores
#SBATCH -t 1-00:00 # Runtime in D-HH:MM
#SBATCH -o JOB%j.out # File to which STDOUT will be written
#SBATCH -e JOB%j.out # File to which STDERR will be written
#SBATCH --mail-type=BEGIN
#SBATCH --mail-user=my@email.com

secs_to_human(){
    echo "$(( ${1} / 3600 )):$(( (${1} / 60) % 60 )):$(( ${1} % 60 ))"
}
start=$(date +%s)
echo "$(date -d @${start} "+%Y-%m-%d %H:%M:%S"): ${SLURM_JOB_NAME} start id=${SLURM_JOB_ID}\n"

### exec task here
( << replace with your task here >> ) \
&& (cat JOB$SLURM_JOB_ID.out |mail -s "$SLURM_JOB_NAME Ended after $(secs_to_human $(($(date +%s) - ${start}))) id=$SLURM_JOB_ID" my@email.com && echo mail sended) \
|| (cat JOB$SLURM_JOB_ID.out |mail -s "$SLURM_JOB_NAME Failed after $(secs_to_human $(($(date +%s) - ${start}))) id=$SLURM_JOB_ID" my@email.com && echo mail sended && exit $?)

you can also edit this to send seperate stdout/stderr logs or attach them as files.

This snippet is also shared on github-gists

elucida
  • 96
  • 5
0

You might also be interested in Slurm-Mail: https://github.com/neilmunday/slurm-mail

Slurm-Mail allows you to customise HTML emails sent to users and provides more information than the emails sent by Slurm by default.

Note: this tool requires admin (root) privileges to install and configure.

MagicUK
  • 44
  • 6
-3

--mail-user=<user> User to receive email notification of state changes as defined by --mail-type . The default value is the submitting user.

And there is an odther parameter is : --mail-type=<type> , it allows you to Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE , INVALID_DEPEND , STAGE_OUT

AissamR38
  • 5
  • 4