Ansible search a list and if a match is found append value as a new value to the dict

Question

I've a dictionary that I'm gathering through an API that looks like below,

ok: [localhost] => {
    "filtered_inventory": [
        {
            "Manufacturer": "",
            "Name": "daughtercard 0/0/CPU0",
            "Serial_number": "serial",
            "device_id": 287,
            "part_id": "ASR9001-LC"
        },
        {
            "Manufacturer": "",
            "Name": "module mau 0/0/2/3",
            "Serial_number": "serial",
            "device_id": 287,
            "part_id": "SFP-10G-SR"
        },

I'm also using the below API to get the below values but it's stored as a list

ok: [localhost] => {
"devices": [
    {
        "device_id": 287,
        "hostname": "example.test.com"
    },
    {
        "device_id": 350,
        "hostname": "example.test.se"
    },

What I want to achive is, if device_id is the same in both the list and the dict then append the hostname to the dictionary. I've solved it with the below code but as I'm working with a big dictionary it's highly inefficiant, running it with the below loop takes around 1 hour and eats a ton of memory.

  - name: Merge hostname with inventory
    #no_log: True
    set_fact:
      merged_list: "{{ merged_list|default([]) +
        [{'hostname': item[0].hostname|upper,
        'device_id': item[0].device_id,
        'serial': item[1].Serial_number,
        'Name': item[1].Name,
        'Manufacturer': item[1].Manufacturer,
        'part_id': item[1].part_id}] }}"
    when: "item[0].device_id == item[1].device_id"
    loop: "{{ query('nested', devices, filtered_inventory) }}"

I've so far been thinking that this probably could be solved by only looping through the dictionary and then match it to the list but haven't figured out how to do that, any ideas?

I'm thinking the below code could be a start but I've tried a few different approaches but failed to get it to work.

  - name: Merge hostname with inventory
    #no_log: True
    set_fact:
      merged_list: "{{ merged_list|default([]) + [item|combine
        ({'device_id': item.device_id,
        'serial': item.Serial_number,
        'Name': item.Name,
        'Manufacturer': item.Manufacturer,
        'part_id': item.part_id,
        'hostname': '' })] }}"
    loop: "{{ filtered_inventory }}"
    when:
      - "item.device_id is in devices|json_query(query)"
    vars:
      query: "'[*].device_id'"

``filtered_inventory`` is also a list, not a dictionary. The first statement should read ``I've dictionaries in the list ...`` perhaps? — Vladimir Botka, May 03 '21 at 09:55

Vladimir Botka · Accepted Answer · 2021-05-05T06:53:07.293

2

Create a dictionary of the hostnames, e.g.

    - set_fact:
        devices_dict: "{{ devices|items2dict(key_name='device_id', value_name='hostname') }}"

gives

  devices_dict:
    287: example.test.com
    350: example.test.se

Then, use this dictionary to combine items of a modified list, e.g.

    - set_fact:
        fi2: "{{ fi2|default([]) +
                 [item|combine({'hostname': devices_dict[item.device_id]})] }}"
      loop: "{{ filtered_inventory }}"

gives

  fi2:
  - Manufacturer: ''
    Name: daughtercard 0/0/CPU0
    Serial_number: serial
    device_id: 287
    hostname: example.test.com
    part_id: ASR9001-LC
  - Manufacturer: ''
    Name: module mau 0/0/2/3
    Serial_number: serial
    device_id: 287
    hostname: example.test.com
    part_id: SFP-10G-SR

Q: "It takes around 6 minutes to run ..."

A: This is too much. Try the filter below

shell> cat filter_plugins/combine_attr.py
def combine_attr(l, d, k, v):
    x = []
    for i in l:
        i[k] = d[i[v]]
        x.append(i)
    return x

class FilterModule(object):
    ''' Ansible filter. Add an attribute to the dictionaries in the list.'''

    def filters(self):
        return {
            'combine_attr': combine_attr,
            }

For example

    - set_fact:
        fi2: "{{ filtered_inventory|
                 combine_attr(devices_dict, 'hostname', 'device_id') }}"

edited May 05 '21 at 06:53

answered May 03 '21 at 09:50

Vladimir Botka

58,131
4
32
63

I was thinking about converting it to a dict and then do the comparison but thought it would be easier with a list but that just explains how much more I've to learn using ansible, thank you for the assistance! – smenk May 03 '21 at 10:25
You're welcome! I'm curious, you mentioned big dictionary ``takes around 1 hour``. How faster is this? Approximately. – Vladimir Botka May 03 '21 at 13:53
It takes around 25 minutes to run through it with the improvements, also the memory usage has gone down by a lot so this is much faster – smenk May 04 '21 at 06:40
This is too much. The Python filter should be faster. – Vladimir Botka May 04 '21 at 08:27
Sorry if I was a little unclear, I meant the whole playbook. The improved loop by itself however took around 6 minutes to run which is more than acceptable so it's a huge improvement. I never timed the old loop but doing some fast math it probably took around 40 minutes.Thanks again! – smenk May 05 '21 at 06:50

Ansible search a list and if a match is found append value as a new value to the dict

1 Answers1