0

I would like to automatically set the state of a node in a SLURM cluster before/after running my Ansible playbook (from idle to drained and after applying the playbook back to idle). The scontrol command that is required for this, is only available on the head node of the cluster. The Ansible playbook, however, is applied to the compute nodes of the cluster. Is there any way to run a remote command on another host than the one currently connected to? I could think of just using the built-in shell module and then SSH to the head node. But maybe there's a nicer way of doing it?

I already looked for ready-to-go Ansible modules but couldn't find any for my use case. The existing ones are all focused on installing/configuring a SLURM cluster.

My idea is then to use a do-until loop that sets the new cluster node state and then repeatedly checks whether the node already switched to the new state (as there still could be running jobs).

Patrick
  • 121
  • 5
  • I don't know SLURM, do you reach the head node also via Ansible? If yes - does the script "know" about the different roles of the nodes in the cluster? If no - can you read the infomation - who is the head node from one of the cluster nodes? Use it as a variable and "delegateTo" the machine. Or - if you can reach the head node from the cluster node use some bad shell-pipes. – TRW Jan 13 '21 at 14:36
  • @TRW Thanks for your comment! Meanwhile I found a solution and updated this post. Yes, I reach the head node via SSH. The script does not know about this as I don't change the head node via Ansible (our admin does though). I hard-coded the head node in my solution. Yes, SLURM provides commands to read this out. I tried `delegateTo` but as far as I remember it runs the command on the node and not locally, also I need to drain the node before running anything on it. – Patrick Jan 13 '21 at 14:50

1 Answers1

0

I finally managed it in the following way:

  • Note that I'm running the playbook from my local client (not on the head node itself).

  • I added a strategy: free command to allow execution on each node as fast as it can because it is less likely that all of the nodes are available at the same time to apply the configuration on.

  • I added a pre_task that is executed before and drains a node:

  pre_tasks:
  - name:
    tags:
    - always
    delegate_to: 127.0.0.1
    shell: ssh -t user@slurm-head "srun -w {{ slurm_node_name}} sleep 3 && echo {{ ldap_passwd }} | sudo -S scontrol update NodeName={{ slurm_node_name }} State=Drain Reason=Maintenance"
    any_errors_fatal: true
  • I added a block that bundles all my tasks and ends with a always section to ensure that even if some task fails, the node is moved back into the Idle status. My assumption here is that failing to apply a configuration doesn't break the node and it's better to make it available again than to block it until I find the time to look into the issue.
  tasks:
    - name: Configure node for Hardware Security course
      block:
      - name: Tasks 1
      ...
      - name: Task 2
      ...

      any_errors_fatal: true
      always:
      - name:
        tags:
        - always
        delegate_to: 127.0.0.1
        shell: ssh -t user@slurm-head "echo {{ ldap_password }} | sudo -S scontrol update NodeName={{ slurm_node_name }} State=RESUME"
Patrick
  • 121
  • 5