3

I observe that when I run a SLURM job, it could create files on other folder paths and also could remove them. It seems dangerous that via SLURM job they can access others folders/files and make changes on them.

$ sbatch run.sh

run.sh:

#!/bin/bash
#SBATCH -o slurm.out        # STDOUT
#SBATCH -e slurm.err        # STDERR
    
echo hello > ~/completed.txt
rm ~/completed.txt

Is it possible to force SLURM to only have access to its own running folder and not others?

alper
  • 2,919
  • 9
  • 53
  • 102

1 Answers1

2

Files access is controlled through UNIX permissions, so a job can only write where the submitting user has permission to write. And most often, a job will need to read and write from and to several distinct directories on distinct filesystems (home NFS for configuration files and results, scratch parallel filesystem for intermediary data and input data, etc.) so Slurm should not confine the job in the submission directory.

If you want to make sure your job has no way to write outside of a specific directory, you can use the chroot command in your job submission script, but that seems a bit odd and less easy to manage than UNIX permissions.

damienfrancois
  • 52,978
  • 9
  • 96
  • 110
  • 1
    If `slurmName` is root: it seems it can make change on any file even submitted user is just a guest user. @damienfrancois – alper Jun 26 '17 at 11:50
  • 1
    Are-you talking about ``SlurmUser``? – damienfrancois Jun 26 '17 at 11:52
  • 1
    Yes sir, the one on the `.conf` file.@damienfrancois – alper Jun 26 '17 at 12:10
  • 1
    That one refers to the name of the user running the controller daemon, on the controller node, not the `slurmd` daemons that run the jobs on the compute nodes – damienfrancois Jun 26 '17 at 12:12
  • 1
    Do you have any example related to use `chroot`? For example I do not want submitted job to remove folder outside of its submitted folder directory. Since it has submitted with guest user it can just do `rm /home/guest` and remove all directories under guest user as I understand. @damienfrancois – alper Jun 26 '17 at 12:37
  • 1
    I guess my best option is to give them guest user permission, if I give my user's permission to them, if they do `rm -f /home/user/*` they may delete my files as well. @damienfrancois – alper May 01 '18 at 12:10
  • 1
    I have make some updates based on your answer. Could you please have a look? – alper May 08 '18 at 16:18
  • 1
    Make sure newUser exists on all compute nodes, and have a look at the slurmctld logs to see why the job failed – damienfrancois May 09 '18 at 06:17
  • 1
    Currently, I have a single compute node so that part shouldn't be problem. Please see the slurmctld.log (https://gist.github.com/avatar-lavventura/322aad1aebdeafec33b9e24cf3ba83c5) . I couldn't understand what might the reason for failure. @damienfrancois – alper May 09 '18 at 09:36
  • 1
    your log file mentions 2 nodes: ` found 2 usable nodes from config containing ebloc[1-2]`. Also this line `REQUEST_JOB_REQUEUE from uid=0` suggests root might have manually requested requeue of the job? – damienfrancois May 09 '18 at 10:31
  • 1
    Please see: `slurm.conf` (https://gist.github.com/avatar-lavventura/46b56cd3a29120594773ae1c8bc4b72c) and `slurmdbd.conf` (https://gist.github.com/avatar-lavventura/a30b0c739cef25430ac998a2299caa8a) . Please note that if the user is `alper`, I can successfully sent the job into slurm (`SlurmUser=alper` on `slurmdbd.con`), but not sure I cannot sent jobs under other user names. ps: Thank you for your valuable help. @damienfrancois – alper May 09 '18 at 11:14
  • 1
    You define 2 nodes both with same 127.0.0.1 IP. Why? Do you have two slurmd daemon running on the same node? That would be a very specific configuration used only for large-scale testing. I would bet that might upset Slurm in your case – damienfrancois May 09 '18 at 11:33
  • 1
    I have only a single machine, I am using slurm's emulation-mode on that machine. I have configured as follows: `./configure --enable-debug --enable-front-end && make install`. But I can try without emulation mode, if I can manage to run slurm. @damienfrancois – alper May 09 '18 at 11:37
  • 1
    Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/170681/discussion-between-damienfrancois-and-alper). – damienfrancois May 09 '18 at 11:39