Fan out to other hosts when already using Serial in ansible

Question

I have 80+ hosts that run my application, and I'm updating a long existing ansible playbook to change our load balancer. In our current load balancer setup, hosts can be added / removed from the load balancer in one ansible play by shelling out to the AWS CLI. However, we're switching to a load balancer configured on a handful of our own hosts, and we will take hosts in and out by manipulating text files on those hosts using ansible. Essentially, I need an inner loop over different hosts within a playbook, while using Serial.

I'm having trouble structuring the playbook such that I can fan out blockinfile commands to hosts in group tag_Type_edge while deploying to the 80 tag_Type_app hosts with serial: 25%.

Here's what I want to be able to do:

---
- hosts: tag_Type_app
  serial: "25%"
  pre_tasks:

    - name: Gathering ec2 facts
      action: ec2_metadata_facts

    - name: Remove from load balancers
      debug:
        msg: "This is where I'd fan out to multiple different hosts from group tag_Type_edge to manipulate
              text files to remove the 25% of hosts from tag_Type_app from the load balancer"

  tasks:
    - name: Do a bunch of work to upgrade the app on the tag_Type_app machines while out of the load balancer
      debug:
        msg: "deploy new code, restart service"


  post_tasks:
    - name: Put back in load balancer
        debug:
          msg: "This is where I'd fan out to multiple different hosts from group tag_Type_edge to manipulate
                 text files to *add* the 25% of hosts from tag_Type_app back into the load balancer"

How can I structure this to allow for the inner loop over tag_Type_edge while using serial: 25% on all the tag_Type_app boxes?

Answer 1

In Ansible, actually running a task on one host, but on behalf of another, could call for the use of a clever thing called delegation. An example in the docs indeed is taking hosts out of a load balancer.

As with any task, it can be run multiple times in a loop on some host names. Not the play loop, but a task loop. An example helps, adopting your skeleton:

---
- name: Example loop over load balancers for each app server
  hosts: tag_Type_app
  serial: "25%"

  pre_tasks:
    - name: Gathering ec2 facts
      action: ec2_metadata_facts

    - name: Remove from load balancers
      debug:
        msg: "Remove app instance {{ inventory_hostname }} from edge node {{ item }}"
      delegate_to: "{{ item }}"
      loop: "{{ query('inventory_hostnames', 'tag_Type_edge') }}"

  tasks:
    - name: Do a bunch of work to upgrade the app on the tag_Type_app machines while out of the load balancer
      debug:
        msg: "deploy new code, restart service"

  post_tasks:
    - name: Put back in load balancer
      debug:
        msg: "Add app instance {{ inventory_hostname }} to edge node {{ item }}"
      delegate_to: "{{ item }}"
      loop: "{{ query('inventory_hostnames', 'tag_Type_edge') }}"

inventory_hostnames is a lookup plugin that does inventory patterns, any pattern can go there.

Couple tricky things about this. Any failure in the "edge" loop will cause the "app" host to fail. Which will leave a partial state of some edge hosts enabled and some not. Unless you have some back out mechanism, like a block with a rescue that undoes it.

As a task loop, some inventory and play related features will not apply. You cannot restrict the edge hosts further with --limit on the command line. Nor can you use serial to batch them up, this is already in a play.

Also it will run a relatively large number of tasks, at least app ✕ edge ✕ 2. This may be a bit slow. Could mitigate somewhat by increasing forks.

If your multiple host load balancer had a single control plane, would not need to touch so many hosts. Doing one task per app server. You might not be set up for this at the moment, but something to consider.