Ansible playbook starting with large forked host list, but spawning serial tasks for smaller subsets of hosts

Question

I'm using Ansible 2.9 and AWX 11. I have a fleet of servers, some of which can be logically segmented by a tag that follows a pattern like GUID_1234567890. There is an arbitrary number of servers represented by each unique GUID_* tag. In addition to the GUID_* tag, a server will be tagged with either foo or bar.

In more concrete terms, I have 2,000 servers, 1,000 of which are tagged with 80 unique GUID_* tags. Within each of those 80 unique GUID_* groups of servers, I have to restart the servers tagged with foo, but not the servers tagged with bar. Furthermore, I have to reboot the foo servers serially if they have the same GUID_* tag.

I can launch a playbook with a host pattern like this: hosts: GUID_*:&foo and get all of the GUID_ tagged foo hosts. However, if the playbook's serial: option is anything except 1, foo servers will be restarted in parallel, most likely multiple foo servers within the same GUID_ tag group. When performed serially, the playbook can take upwards of a day to complete. I'd like to use Ansible's forking to run a fork of a playbook for each of the 80 groups of GUID_* hosts, but each fork must call a playbook that itself runs as serial:1 against the foo servers.

Is there a way to start with large set of hosts, but then create parallel workers that execute serial: 1 playbooks for a subset of hosts?

Answer 1

It's not possible to achieve the goal in a single playbook. A dedicated playbook for each group is needed if you want to process the groups in parallel. Use Ansible to create the code.

"Smaller subsets of hosts"

Given the inventory to simplify the testing

test_01
test_02
test_03
test_04
test_05
test_06

[GUID_01]
test_01
test_02
test_03

[GUID_02]
test_04
test_05
test_06

[foo]
test_01
test_02
test_04
test_05
srvX

[bar]
test_03
test_06
srvY

Create a dictionary of all GUID_* and a list of hosts both in particular GUID_x and foo groups. For example

- hosts: GUID_*:&foo
  gather_facts: false
  tasks:
    - set_fact:
        my_groups: "{{ my_groups|default({})|
                       combine({item: my_hosts|dict2items|
                                      selectattr('value', 'eq', item)|
                                      map(attribute='key')|list}) }}"
      loop: "{{ my_guids|unique }}"
      vars:
        my_guids: "{{ ansible_play_hosts_all|
                      map('extract', hostvars, 'group_names')|
                      map('select', 'match', 'GUID')|
                      map('first')|flatten }}"
        my_hosts: "{{ dict(ansible_play_hosts_all|zip(my_guids)) }}"
      run_once: true
    - debug:
        var: my_groups
      run_once: true

gives

  my_groups:
    GUID_01:
    - test_01
    - test_02
    GUID_02:
    - test_04
    - test_05

---

The same dictionary can be created without patterns

- hosts: all
  gather_facts: false
  tasks:
    - set_fact:
        my_groups: "{{ my_groups|default({})|
                       combine({item: groups[item]|
                                      intersect(groups['foo'])}) }}"
      loop: "{{ groups|select( 'match', 'GUID')|list }}"
      run_once: true

---

Q: "Create parallel workers that execute serial: 1 playbooks for a subset of hosts."

A: Use Ansible templates to create the code. For example, given the role

shell> cat roles/reboot/tasks/main.yml
- command: date "+%H:%M:%S"
  register: result
- debug:
    msg: "{{ result.stdout }} {{ inventory_hostname }} Reboot"
- command: sleep 3
- command: date "+%H:%M:%S"
  register: result
- debug:
    msg: "{{ result.stdout }} {{ inventory_hostname }} Ready"

The templates

shell> cat templates/my_hosts.j2
{{ lookup('file', 'hosts') }}

# ---------------------------------------------------------------------
# Groups for my_wrapper. See template my_hosts.j2

{% for group,hosts in my_groups.items() %}
[my_{{ group }}]
{% for host in hosts %}
{{ host }}
{% endfor %}

{% endfor %}

shell> cat templates/my_pb.j2
- hosts: my_{{ item }}
  gather_facts: false
  serial: 1
  roles:
    - reboot

shell> cat templates/my_wrapper.j2
#!/bin/bash
{% for group,hosts in my_groups.items() %}
nohup ansible-playbook -i my_hosts my_pb_{{ group }}.yml &
{% endfor %}

and the playbook

- hosts: all
  gather_facts: false
  tasks:
    - set_fact:
        my_groups: "{{ my_groups|default({})|
                       combine({item: groups[item]|
                                      intersect(groups['foo'])}) }}"
      loop: "{{ groups|select( 'match', 'GUID')|list }}"
      run_once: true
    - block:
        - template:
            src: my_hosts.j2
            dest: my_hosts
        - template:
            src: my_wrapper.j2
            dest: my_wrapper.sh
            mode: "a+x"
        - template:
            src: my_pb.j2
            dest: "my_pb_{{ item }}.yml"
          loop: "{{ my_groups.keys()|list }}"
      run_once: true
      delegate_to: localhost

create the files with the code

shell> cat my_hosts

   ...

# ---------------------------------------------------------------------
# Groups for my_wrapper. See template my_hosts.j2

[my_GUID_01]
test_01
test_02

[my_GUID_02]
test_04
test_05

shell> cat my_wrapper.sh
#!/bin/bash
nohup ansible-playbook -i my_hosts my_pb_GUID_01.yml &
nohup ansible-playbook -i my_hosts my_pb_GUID_02.yml &

shell> cat my_pb_GUID_01.yml 
- hosts: my_GUID_01
  gather_facts: false
  serial: 1
  roles:
    - reboot

shell> cat my_pb_GUID_02.yml 
- hosts: my_GUID_02
  gather_facts: false
  serial: 1
  roles:
    - reboot

Then, running the wrapper gives

shell> grep msg nohup.out 
    "msg": "13:03:51 test_01 Reboot"
    "msg": "13:03:51 test_04 Reboot"
    "msg": "13:03:56 test_01 Ready"
    "msg": "13:03:56 test_04 Ready"
    "msg": "13:03:58 test_02 Reboot"
    "msg": "13:03:58 test_05 Reboot"
    "msg": "13:04:04 test_02 Ready"
    "msg": "13:04:04 test_05 Ready"