1

I am trying to create a ansible role which filter out host based on dictionary(passed to role) in comparison with host variables. If host variables contains key:value from dictionary it will added to filtered_host list. This is what I found till point but not getting desired output:

Ansible role:

---

- name: Filtering host
  debug:
    msg: "Here is the response {{ filters }} {{groups['all']}}"

- name: Setting filter
  set_fact:
    input_filter: "{{ filters }}"
  
- name: Setting filtered hosts
  set_fact:
    filtered_hosts: "{{ groups['all'] | map('extract', hostvars) | select('@ in filters') | map(attribute='inventory_hostname') | list }}"
  vars:
    input_filter: "{{ filters | to_json }}"

- name: Print filtered hosts
  debug:
    var: filtered_hosts

Here filters is a dictionary(filter) passed from playbook to test but actually i want to pass a dictionary, and this role should return list of hosts which contains key:value from this dictionary.

Sample dictionary:

{
   "os_type":"linux",
   "datacenter" : "REM",
   "location" : "IND"
}

Sample Host variables:

HOST A variables:

{
   "os_type":"linux",
   "datacenter" : "REM",
   "location" : "IND"
}

HOST B variables:

{
   "os_type":"linux",
   "datacenter" : "REM",
   "location" : "USA",
   "status" : "success"
}

HOST C variables:

{
   "os_type":"linux",
   "datacenter" : "REM",
   "location" : "IND",
   "status" : "success"
}

OUTPUT:

[Host A, Host C]

because both host contains key:value passed through dictionary. Also number of keys differ from different hosts.

Actual Scenario: I created a dynamic inventory which will fetch all host and host vars from DB based on some filters and assign it to inventory. Groups are created using some general info based on host vars. Now templates are created for different organizations/product who require to filter host based on some conditions(dynamically) out of the host vars. Since we cant create groups for everything, we decided to have a common ansible role which will act as a filter.

Within playbook there are multiple task will be performed on different host and to get those specific host some conditions are generated in the form of dictionary. These dictionary will be passed to ansible role to filter further based on the dynamic condition. The filter in given scenario is comparing 2 dictionaries to find out proper host for that task. A simplified example is what I have shown in the question.

Zeitounator
  • 38,476
  • 7
  • 53
  • 66
DrunkenAvenger
  • 37
  • 11
  • It looks like your are basically stuck in an [x/y problem](https://xyproblem.info). Can you please [edit] and give more information on what you are trying to achieve exactly? My understanding at this point is that you are trying to reinvent [dynamic inventory](https://docs.ansible.com/ansible/latest/inventory_guide/intro_dynamic_inventory.html) and [host limit](https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html#cmdoption-ansible-playbook-l). – Zeitounator Aug 07 '23 at 10:31
  • No @Zeitounator i already setup'd dynamic inventory but while running task i need one more level of filter based on customer requirement such as datacenter, os type etc which will be generated as part of playbook run/task, so this output(dictionary) will be passed to role to filter host again. – DrunkenAvenger Aug 07 '23 at 10:35
  • If you have a dynamic inventory, you can set groups by DC and use patterns to fetch exactly what you need. – Zeitounator Aug 07 '23 at 10:44
  • filters are dynamic and there are multiple customers under different organization, creating group everytime they run tasks is not what they want. Thats why this filter option. – DrunkenAvenger Aug 07 '23 at 10:54
  • Creating hundreds or even thousands of dynamic groups from your inventory based on host vars is not very resource intensive (and can even be cached). You don't have to use them all. Once you have that structure, it's very easy to filter what you need with [patterns](https://docs.ansible.com/ansible/latest/inventory_guide/intro_patterns.html) from your provided dynamic variables. Once again, knowing with a bit more details what you are trying to achieve would help giving an accurate answer. – Zeitounator Aug 07 '23 at 12:14
  • I understand your point @Zeitounator but since things are not in my hand to achieve what you are suggesting, thats why i came up with the question on how we can filter further more based on host vars and filter dictionary. That is the requirement I got to work on. – DrunkenAvenger Aug 07 '23 at 13:10
  • @Zeitounator Can you help me with the filter as of now?? – DrunkenAvenger Aug 07 '23 at 15:35

2 Answers2

1

Given the dictionary

  filters:
    os_type: linux
    datacenter: REM
    location: IND

In the vars, select the keys

  fkeys: "{{ filters.keys() }}"

Select and join the values

  fvals: "{{ filters.values()|join(',') }}"

gives

  fvals: linux,REM,IND

In the tasks, create the variable my_fvals

    - set_fact:
        my_fvals: "{{ lookup('vars', *fkeys, default='UNDEF') }}"
    - debug:
        var: my_fvals

gives (abridged)

ok: [Host_A] => 
  my_fvals: linux,REM,IND
ok: [Host_B] => 
  my_fvals: linux,REM,USA
ok: [Host_C] => 
  my_fvals: linux,REM,IND

Select hosts that match the criteria

  filtered_hosts: "{{ hostvars|dict2items|
                      selectattr('value.my_fvals', '==', fvals)|
                      map(attribute='key') }}"

gives

  filtered_hosts: [Host_A, Host_C]
  • Example of a complete playbook for testing
- hosts: all

  vars:

    filters:
      os_type: linux
      datacenter: REM
      location: IND

    fkeys: "{{ filters.keys() }}"
    fvals: "{{ filters.values()|join(',') }}"
    filtered_hosts: "{{ hostvars|dict2items|
                        selectattr('value.my_fvals', '==', fvals)|
                        map(attribute='key') }}"

  tasks:

    - debug:
        var: fvals
      run_once: true

    - set_fact:
        my_fvals: "{{ lookup('vars', *fkeys, default='UNDEF') }}"
    - debug:
        var: my_fvals

    - debug:
        var: filtered_hosts|to_yaml
      run_once: true
  • You can create a role if you want to
shell> tree roles
roles/
└── filtered_hosts
    ├── defaults
    │   └── main.yml
    └── tasks
        └── main.yml

3 directories, 2 files

shell> cat roles/filtered_hosts/defaults/main.yml 
fkeys: "{{ filters.keys() }}"
fvals: "{{ filters.values()|join(',') }}"

shell> cat roles/filtered_hosts/tasks/main.yml 
- set_fact:
    my_fvals: "{{ lookup('vars', *fkeys, default='UNDEF') }}"
- set_fact:
    filtered_hosts: "{{ hostvars|dict2items|
                        selectattr('value.my_fvals', '==', fvals)|
                        map(attribute='key') }}"
  run_once: true

The playbook

shell> cat pb.yml
- hosts: all

  vars:

    filters:
      os_type: linux
      datacenter: REM
      location: IND

  roles:
    - filtered_hosts

  tasks:

    - debug:
        var: filtered_hosts|to_yaml
      run_once: true

gives

shell> ansible-playbook pb.yml

PLAY [all] ***********************************************************************************

TASK [filtered_hosts : set_fact] *************************************************************
ok: [Host_A]
ok: [Host_C]
ok: [Host_B]

TASK [filtered_hosts : set_fact] *************************************************************
ok: [Host_A]

TASK [debug] *********************************************************************************
ok: [Host_A] => 
  filtered_hosts|to_yaml: |-
    [Host_A, Host_C]

PLAY RECAP ***********************************************************************************
Host_A: ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
Host_B: ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
Host_C: ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
  • It is not possible to use the inventory plugin ansible.builtin.constructed here
shell> tree inventory
inventory/
├── 01-hosts
└── 02-constructed.yml

0 directories, 2 files

shell> cat inventory/01-hosts 
Host_A
Host_B
Host_C

shell> cat inventory/02-constructed.yml 
plugin: ansible.builtin.constructed
use_extra_vars: true
use_vars_plugins: true
strict: true
compose:
  fkeys: filters.keys()
  fvals: filters.values()|join(',')
  my_fvals: lookup('vars', *filters.keys(), default='UNDEF')
groups:
  filtered_hosts: my_fvals == fvals

because in this inventory plugin lookups were disabled from templating

shell> ansible-inventory -i inventory -e @filters.yml --list --yaml

[WARNING]: * Failed to parse /export/scratch/tmp7/test-477/inventory/02-constructed.yml with auto plugin: failed to parse /export/scratch/tmp7/test-477/inventory/02-constructed.yml: Could not set my_fvals for host Host_A: The lookup vars was found, however lookups were disabled from templating . Could not set my_fvals for host Host_A: The lookup vars was found, however lookups were disabled from templating

[WARNING]: * Failed to parse /export/scratch/tmp7/test-477/inventory/02-constructed.yml with yaml plugin: Plugin configuration YAML file, not YAML inventory

[WARNING]: * Failed to parse /export/scratch/tmp7/test-477/inventory/02-constructed.yml with ini plugin: Invalid host pattern 'plugin:' supplied, ending in ':' is not allowed, this character is reserved to provide a port.

[WARNING]: Unable to parse /export/scratch/tmp7/test-477/inventory/02-constructed.yml as an inventory source

all:
  children:
    ungrouped:
      hosts:
        Host_A:
          datacenter: REM
          filters: &id001
            datacenter: REM
            location: IND
            os_type: linux
          fkeys:
          - os_type
          - datacenter
          - location
          fvals: linux,REM,IND
          location: IND
          os_type: linux
        Host_B:
          datacenter: REM
          filters: *id001
          location: USA
          os_type: linux
          status: success
        Host_C:
          datacenter: REM
          filters: *id001
          location: IND
          os_type: linux
          status: success
Vladimir Botka
  • 58,131
  • 4
  • 32
  • 63
  • thanks for the detailed explaination and this works great when considering playbook. But how can we manage to get the filtered_hosts part and my_fvals set_fact part into a ansible role??? The original ask is to have a ANSIBLE ROLE for this filter which return the list of host matching the filters. Can you guide me with this? – DrunkenAvenger Aug 07 '23 at 19:19
  • The filter dictionary should be passed from playbook to role and role in return gives the list of all host based on the filter. – DrunkenAvenger Aug 07 '23 at 19:21
  • I added an example of the role. – Vladimir Botka Aug 07 '23 at 19:33
0

This is your current inventory (as I understand it) "dumped" into a static inventory. This can be replaced with your current dynamic inventory. I placed this in inventories/default/0-hosts.yml

---
all:
  hosts:
    host_a: {
      "os_type": "linux",
      "datacenter": "REM",
      "location": "IND"
    }
    host_b: {
      "os_type": "linux",
      "datacenter": "REM",
      "location": "USA",
      "status": "success"
    }
    host_c: {
      "os_type": "linux",
      "datacenter": "REM",
      "location": "IND",
      "status": "success"
    }

Lets add a new ansible.builtin.constructed source where we create dynamically all the necessary groups from your variables. This goes in inventories/default/1-constructed.yml

---
plugin: ansible.builtin.constructed
strict: false
keyed_groups:
  - prefix: os_type
    key: os_type
  - prefix: datacenter
    key: datacenter
  - prefix: location
    key: location

We can already explore the result from theansible-inventory -i inventories/default/ --list command:

{
    "_meta": {
        "hostvars": {
            "host_a": {
                "datacenter": "REM",
                "location": "IND",
                "os_type": "linux"
            },
            "host_b": {
                "datacenter": "REM",
                "location": "USA",
                "os_type": "linux",
                "status": "success"
            },
            "host_c": {
                "datacenter": "REM",
                "location": "IND",
                "os_type": "linux",
                "status": "success"
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "os_type_linux",
            "datacenter_REM",
            "location_IND",
            "location_USA"
        ]
    },
    "datacenter_REM": {
        "hosts": [
            "host_a",
            "host_b",
            "host_c"
        ]
    },
    "location_IND": {
        "hosts": [
            "host_a",
            "host_c"
        ]
    },
    "location_USA": {
        "hosts": [
            "host_b"
        ]
    },
    "os_type_linux": {
        "hosts": [
            "host_a",
            "host_b",
            "host_c"
        ]
    }
}

From there it is quite easy to write a playbook which will take advantage of this multi-source inventory taking into account your user variables. As a pure example, this is what I tried in playbook.yml. I passed your user filters as extra vars for the example but they can come from any source.

---
- name: Process user vars
  hosts: localhost
  gather_facts: false

  tasks:
    - name: Create a pattern from filter (defaulting to all if filter is empty)
      vars:
        filter_data: "{{ user_filter | d({}) }}"
      ansible.builtin.set_fact:
        filter_pattern: "{{
          filter_data.keys() | zip(filter_data.values())
          | map('join', '_') | join(':&') | d('all', true)
        }}"
    
    - name: Show the constructed pattern
      ansible.builtin.debug:
        var: hostvars.localhost.filter_pattern

    - name: We can now use that pattern in a lookup in any play to get a list
      ansible.builtin.debug:
        var: "q(
          'ansible.builtin.inventory_hostnames',
          hostvars.localhost.filter_pattern
        )"

- name: We can also use the pattern to target hosts in an other play
  hosts: "{{ hostvars.localhost.filter_pattern }}"
  gather_facts: false

  # Note that you could play a role here

  tasks:
    - name: Show inventory hostname we would run on
      ansible.builtin.debug:
        var: inventory_hostname

Demo runs:

  • With your original filter
$ ansible-playbook -i inventories/default/ -e '{"user_filter":{"os_type":"linux", "datacenter":"REM", "location":"IND"}}' playbook.yml 

PLAY [Process user vars] ******************************************************************************************************************************************************************************************

TASK [Create a pattern from filter (defaulting to all if filter is empty)] ****************************************************************************************************************************************
ok: [localhost]

TASK [Show the constructed pattern] *******************************************************************************************************************************************************************************
ok: [localhost] => {
    "hostvars.localhost.filter_pattern": "os_type_linux:&datacenter_REM:&location_IND"
}

TASK [We can now use that pattern in a lookup in any play to get a list] ******************************************************************************************************************************************
ok: [localhost] => {
    "q( 'ansible.builtin.inventory_hostnames', hostvars.localhost.filter_pattern )": [
        "host_a",
        "host_c"
    ]
}

PLAY [We can also use the pattern to target hosts in an other play] ***********************************************************************************************************************************************

TASK [Show inventory hostname we would run on] ********************************************************************************************************************************************************************
ok: [host_a] => {
    "inventory_hostname": "host_a"
}
ok: [host_c] => {
    "inventory_hostname": "host_c"
}

PLAY RECAP ********************************************************************************************************************************************************************************************************
host_a                     : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
host_c                     : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
  • Change location
$ ansible-playbook -i inventories/default/ -e '{"user_filter":{"os_type":"linux", "datacenter":"REM", "location":"USA"}}' playbook.yml 

PLAY [Process user vars] ******************************************************************************************************************************************************************************************

TASK [Create a pattern from filter (defaulting to all if filter is empty)] ****************************************************************************************************************************************
ok: [localhost]

TASK [Show the constructed pattern] *******************************************************************************************************************************************************************************
ok: [localhost] => {
    "hostvars.localhost.filter_pattern": "os_type_linux:&datacenter_REM:&location_USA"
}

TASK [We can now use that pattern in a lookup in any play to get a list] ******************************************************************************************************************************************
ok: [localhost] => {
    "q( 'ansible.builtin.inventory_hostnames', hostvars.localhost.filter_pattern )": [
        "host_b"
    ]
}

PLAY [We can also use the pattern to target hosts in an other play] ***********************************************************************************************************************************************

TASK [Show inventory hostname we would run on] ********************************************************************************************************************************************************************
ok: [host_b] => {
    "inventory_hostname": "host_b"
}

PLAY RECAP ********************************************************************************************************************************************************************************************************
host_b                     : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
  • Defaults to all hosts if no filter is provided
$ ansible-playbook -i inventories/default/ playbook.yml 

PLAY [Process user vars] ******************************************************************************************************************************************************************************************

TASK [Create a pattern from filter (defaulting to all if filter is empty)] ****************************************************************************************************************************************
ok: [localhost]

TASK [Show the constructed pattern] *******************************************************************************************************************************************************************************
ok: [localhost] => {
    "hostvars.localhost.filter_pattern": "all"
}

TASK [We can now use that pattern in a lookup in any play to get a list] ******************************************************************************************************************************************
ok: [localhost] => {
    "q( 'ansible.builtin.inventory_hostnames', hostvars.localhost.filter_pattern )": [
        "host_a",
        "host_b",
        "host_c"
    ]
}

PLAY [We can also use the pattern to target hosts in an other play] ***********************************************************************************************************************************************

TASK [Show inventory hostname we would run on] ********************************************************************************************************************************************************************
ok: [host_a] => {
    "inventory_hostname": "host_a"
}
ok: [host_b] => {
    "inventory_hostname": "host_b"
}
ok: [host_c] => {
    "inventory_hostname": "host_c"
}

PLAY RECAP ********************************************************************************************************************************************************************************************************
host_a                     : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
host_b                     : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
host_c                     : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
Zeitounator
  • 38,476
  • 7
  • 53
  • 66