2

I have 2 lists that I'm trying to combine together, by a specific attribute (the "device" attribute). I am not instantiating the lists in my code anywhere, they are being populated by API calls so the best I have here are printouts of each list.

List1:

TASK [Print List1] **************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": [
        [
            {
                "device": "LTM1_Device",
                "link": "LTM1_Link",
                "ltm_pool": "LTM1_Pool"
            },
            {
                "device": "LTM2_Device",
                "link": "LTM2_Link",
                "ltm_pool": "LTM2_Pool"
            },
            {
                "device": "LTM3_Device",
                "link": "LTM3_Link",
                "ltm_pool": "LTM3_Pool"
            }
        ]
    ]
}

List2:

TASK [Print List2] ****************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": [
        [
            {
                "device": "LTM1_Device",
                "host": "/Common/LTM1_Host1",
                "ip": "0.0.0.1",
                "port": "5555"
            },
            {
                "device": "LTM1_Device",
                "host": "/Common/LTM1_Host2",
                "ip": "0.0.0.2",
                "port": "5555"
            },
            {
                "device": "LTM2_Device",
                "host": "/Common/LTM2_Host1",
                "ip": "0.0.0.3",
                "port": "5555"
            },
            {
                "device": "LTM2_Device",
                "host": "/Common/LTM2_Host2",
                "ip": "0.0.0.4",
                "port": "5555"
            },
            {
                "device": "LTM3_Device",
                "host": "/Common/LTM3_Host1",
                "ip": "0.0.0.5",
                "port": "5555"
            },
            {
                "device": "LTM3_Device",
                "host": "/Common/LTM3_Host2",
                "ip": "0.0.0.6",
                "port": "5555"
            }
        ]
    ]
}

I have 2 debug lines that print something fairly close to what I am looking for, however neither are quite what I want... and I can't seem to get the Jinja2 filter correct.

- name: Debug
    debug:
        msg: 
            - "{{ (list2 + list1) | groupby('device') | map('last') | map('combine') | list }}"
            - "{{ (list2 + list1) | groupby('device') | map('last') | list }}"

Which gives the following output:

TASK [Debug] ****************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": [
        [
            {
                "device": "LTM1_Device",
                "host": "/Common/LTM1_Host2",
                "ip": "0.0.0.1",
                "link": "LTM1_Link",
                "ltm_pool": "LTM1_Pool",
                "port": "5555"
            },
            {
                "device": "LTM2_Device",
                "host": "/Common/LTM2_Host2",
                "ip": "0.0.0.3",
                "link": "LTM2_Link",
                "ltm_pool": "LTM2_Pool",
                "port": "5555"
            },
            {
                "device": "LTM3_Device",
                "host": "/Common/LTM2_Host2",
                "ip": "0.0.0.5",
                "link": "LTM3_Link",
                "ltm_pool": "LTM3_Pool",
                "port": "5555"
            }
        ],
        [
            [
                {
                    "device": "LTM1_Device",
                    "host": "/Common/LTM1_Host1",
                    "ip": "0.0.0.1",
                    "port": "5555"
                },
                {
                    "device": "LTM1_Device",
                    "host": "/Common/LTM1_Host2",
                    "ip": "0.0.0.2",
                    "port": "5555"
                },
                {
                    "device": "LTM1_Device",
                    "link": "LTM1_Link",
                    "ltm_pool": "LTM1_Pool"
                }
            ],
            [
                {
                    "device": "LTM2_Device",
                    "host": "/Common/LTM2_Host1",
                    "ip": "0.0.0.3",
                    "port": "5555"
                },
                {
                    "device": "LTM2_Device",
                    "host": "/Common/LTM2_Host2",
                    "ip": "0.0.0.4",
                    "port": "5555"
                },
                {
                    "device": "LTM2_Device",
                    "link": "LTM2_Link",
                    "ltm_pool": "LTM2_Pool"
                }
            ],
            [
                {
                    "device": "LTM3_Device",
                    "host": "/Common/LTM3_Host1",
                    "ip": "0.0.0.5",
                    "port": "5555"
                },
                {
                    "device": "LTM3_Device",
                    "host": "/Common/LTM3_Host2",
                    "ip": "0.0.0.6",
                    "port": "5555"
                },
                {
                    "device": "LTM3_Device",
                    "link": "LTM3_Link",
                    "ltm_pool": "LTM3_Pool"
                }
            ]
        ]
    ]
}

The first debug line does not combine both lists for all elements of list2 (it misses the first host for each LTM), and the second line combines them as a separate element in a nested list. Is there a way to combine both lists so it would look something like the following? I know I'm probably missing something small but I can't seem to figure out what it is I'm missing. Any help would be greatly appreciated, thanks!

Desired output (I'm not worried about it being a nested list as I can flatten that later):

        [
            [
                {
                    "device": "LTM1_Device",
                    "host": "/Common/LTM1_Host1",
                    "ip": "0.0.0.1",
                    "link": "LTM1_Link",
                    "ltm_pool": "LTM1_Pool"
                    "port": "5555"
                },
                {
                    "device": "LTM1_Device",
                    "host": "/Common/LTM1_Host2",
                    "ip": "0.0.0.2",
                    "link": "LTM1_Link",
                    "ltm_pool": "LTM1_Pool"
                    "port": "5555"
                }
            ],
            [
                {
                    "device": "LTM2_Device",
                    "host": "/Common/LTM2_Host1",
                    "ip": "0.0.0.3",
                    "link": "LTM2_Link",
                    "ltm_pool": "LTM2_Pool"
                    "port": "5555"
                },
                {
                    "device": "LTM2_Device",
                    "host": "/Common/LTM2_Host2",
                    "ip": "0.0.0.4",
                    "link": "LTM2_Link",
                    "ltm_pool": "LTM2_Pool"
                    "port": "5555"
                }
            ],
            [
                {
                    "device": "LTM3_Device",
                    "host": "/Common/LTM3_Host1",
                    "ip": "0.0.0.5",
                    "link": "LTM3_Link",
                    "ltm_pool": "LTM3_Pool"
                    "port": "5555"
                },
                {
                    "device": "LTM3_Device",
                    "host": "/Common/LTM3_Host2",
                    "ip": "0.0.0.6",
                    "link": "LTM3_Link",
                    "ltm_pool": "LTM3_Pool"
                    "port": "5555"
                }
            ]
        ]
cilles
  • 21
  • 3

2 Answers2

1

There are more options.

  • Iterate the lists if list1 is properly sorted and provides exactly what you need
    - set_fact:
        result: "{{ result|d([]) + _item }}"
      with_together:
        - "{{ list1.0 }}"
        - "{{ list2.0|groupby('device')|map(attribute=1)|list }}"
      vars:
        _item: "{{ [item.0]|product(item[1:])|map('combine')|list }}"

gives the desired result

result:
  - device: LTM1_Device
    host: /Common/LTM1_Host1
    ip: 0.0.0.1
    link: LTM1_Link
    ltm_pool: LTM1_Pool
    port: '5555'
  - device: LTM1_Device
    host: /Common/LTM1_Host2
    ip: 0.0.0.2
    link: LTM1_Link
    ltm_pool: LTM1_Pool
    port: '5555'
  - device: LTM2_Device
    host: /Common/LTM2_Host1
    ip: 0.0.0.3
    link: LTM2_Link
    ltm_pool: LTM2_Pool
    port: '5555'
  - device: LTM2_Device
    host: /Common/LTM2_Host2
    ip: 0.0.0.4
    link: LTM2_Link
    ltm_pool: LTM2_Pool
    port: '5555'
  - device: LTM3_Device
    host: /Common/LTM3_Host1
    ip: 0.0.0.5
    link: LTM3_Link
    ltm_pool: LTM3_Pool
    port: '5555'
  - device: LTM3_Device
    host: /Common/LTM3_Host2
    ip: 0.0.0.6
    link: LTM3_Link
    ltm_pool: LTM3_Pool
    port: '5555'

(details)

  • The next option is converting list1 to dictionaries
links: "{{ list1.0|items2dict(key_name='device', value_name='link') }}"
pools: "{{ list1.0|items2dict(key_name='device', value_name='ltm_pool') }}"

give

links:
  LTM1_Device: LTM1_Link
  LTM2_Device: LTM2_Link
  LTM3_Device: LTM3_Link
pools:
  LTM1_Device: LTM1_Pool
  LTM2_Device: LTM2_Pool
  LTM3_Device: LTM3_Pool

Iterate list2 and combine the dictionaries. The task below gives also the desired result

    - set_fact:
        result: "{{ result|d([]) + [_item] }}"
      loop: "{{ list2.0 }}"
      vars:
        _item: "{{ item|combine({'link': links[item.device]})|
                        combine({'ltm_pool': pools[item.device]}) }}"

(details)

Notes

  1. Example of a complete playbook
- hosts: localhost
  vars:
    list1:
      - - device: LTM1_Device
          link: LTM1_Link
          ltm_pool: LTM1_Pool
        - device: LTM2_Device
          link: LTM2_Link
          ltm_pool: LTM2_Pool
        - device: LTM3_Device
          link: LTM3_Link
          ltm_pool: LTM3_Pool
    list2:
      - - device: LTM1_Device
          host: /Common/LTM1_Host1
          ip: 0.0.0.1
          port: '5555'
        - device: LTM1_Device
          host: /Common/LTM1_Host2
          ip: 0.0.0.2
          port: '5555'
        - device: LTM2_Device
          host: /Common/LTM2_Host1
          ip: 0.0.0.3
          port: '5555'
        - device: LTM2_Device
          host: /Common/LTM2_Host2
          ip: 0.0.0.4
          port: '5555'
        - device: LTM3_Device
          host: /Common/LTM3_Host1
          ip: 0.0.0.5
          port: '5555'
        - device: LTM3_Device
          host: /Common/LTM3_Host2
          ip: 0.0.0.6
          port: '5555'
    links: "{{ list1.0|items2dict(key_name='device', value_name='link') }}"
    pools: "{{ list1.0|items2dict(key_name='device', value_name='ltm_pool') }}"
  tasks:
    - set_fact:
        result: "{{ result|d([]) + [_item] }}"
      loop: "{{ list2.0 }}"
      vars:
        _item: "{{ item|combine({'link': links[item.device]})|
                        combine({'ltm_pool': pools[item.device]}) }}"
    - debug:
        var: result
  1. If you want to avoid iteration create lists of the attributes ip and device
ip: "{{ list2.0|map(attribute='ip')|list }}"
device: "{{ list2.0|map(attribute='device')|list }}"

give

ip: [0.0.0.1, 0.0.0.2, 0.0.0.3, 0.0.0.4, 0.0.0.5, 0.0.0.6]
device: [LTM1_Device, LTM1_Device, LTM2_Device, LTM2_Device, LTM3_Device, LTM3_Device]

Then use the list device and extract both links and pools

link: "{{ device|map('extract', links)|list }}"
pool: "{{ device|map('extract', pools)|list }}"

give

link: [LTM1_Link, LTM1_Link, LTM2_Link, LTM2_Link, LTM3_Link, LTM3_Link]
pool: [LTM1_Pool, LTM1_Pool, LTM2_Pool, LTM2_Pool, LTM3_Pool, LTM3_Pool]

Create dictionaries

link_dict: "{{ dict(ip|zip(link)) }}"
pool_dict: "{{ dict(ip|zip(pool)) }}"

give

link_dict:
  0.0.0.1: LTM1_Link
  0.0.0.2: LTM1_Link
  0.0.0.3: LTM2_Link
  0.0.0.4: LTM2_Link
  0.0.0.5: LTM3_Link
  0.0.0.6: LTM3_Link
pool_dict:
  0.0.0.1: LTM1_Pool
  0.0.0.2: LTM1_Pool
  0.0.0.3: LTM2_Pool
  0.0.0.4: LTM2_Pool
  0.0.0.5: LTM3_Pool
  0.0.0.6: LTM3_Pool

and convert the dictionaries to lists

link_list: "{{ link_dict|dict2items(key_name='ip', value_name='link') }}"
pool_list: "{{ pool_dict|dict2items(key_name='ip', value_name='ltm_pool') }}"

give

link_list:
  - {ip: 0.0.0.1, link: LTM1_Link}
  - {ip: 0.0.0.2, link: LTM1_Link}
  - {ip: 0.0.0.3, link: LTM2_Link}
  - {ip: 0.0.0.4, link: LTM2_Link}
  - {ip: 0.0.0.5, link: LTM3_Link}
  - {ip: 0.0.0.6, link: LTM3_Link}
pool_list:
  - {ip: 0.0.0.1, ltm_pool: LTM1_Pool}
  - {ip: 0.0.0.2, ltm_pool: LTM1_Pool}
  - {ip: 0.0.0.3, ltm_pool: LTM2_Pool}
  - {ip: 0.0.0.4, ltm_pool: LTM2_Pool}
  - {ip: 0.0.0.5, ltm_pool: LTM3_Pool}
  - {ip: 0.0.0.6, ltm_pool: LTM3_Pool}

Finally, use filter community.general.lists_mergeby and merge the lists by attribute ip. This gives the desired result

result: "{{ [list2.0, link_list, pool_list]|community.general.lists_mergeby('ip') }}"

Example of a complete playbook

- hosts: localhost
  vars:
    list1:
      - - device: LTM1_Device
          link: LTM1_Link
          ltm_pool: LTM1_Pool
        - device: LTM2_Device
          link: LTM2_Link
          ltm_pool: LTM2_Pool
        - device: LTM3_Device
          link: LTM3_Link
          ltm_pool: LTM3_Pool
    list2:
      - - device: LTM1_Device
          host: /Common/LTM1_Host1
          ip: 0.0.0.1
          port: '5555'
        - device: LTM1_Device
          host: /Common/LTM1_Host2
          ip: 0.0.0.2
          port: '5555'
        - device: LTM2_Device
          host: /Common/LTM2_Host1
          ip: 0.0.0.3
          port: '5555'
        - device: LTM2_Device
          host: /Common/LTM2_Host2
          ip: 0.0.0.4
          port: '5555'
        - device: LTM3_Device
          host: /Common/LTM3_Host1
          ip: 0.0.0.5
          port: '5555'
        - device: LTM3_Device
          host: /Common/LTM3_Host2
          ip: 0.0.0.6
          port: '5555'
    links: "{{ list1.0|items2dict(key_name='device', value_name='link') }}"
    pools: "{{ list1.0|items2dict(key_name='device', value_name='ltm_pool') }}"
    ip: "{{ list2.0|map(attribute='ip')|list }}"
    device: "{{ list2.0|map(attribute='device')|list }}"
    link: "{{ device|map('extract', links)|list }}"
    pool: "{{ device|map('extract', pools)|list }}"
    link_dict: "{{ dict(ip|zip(link)) }}"
    pool_dict: "{{ dict(ip|zip(pool)) }}"
    link_list: "{{ link_dict|dict2items(key_name='ip', value_name='link') }}"
    pool_list: "{{ pool_dict|dict2items(key_name='ip', value_name='ltm_pool') }}"
    result: "{{ [list2.0, link_list, pool_list]|community.general.lists_mergeby('ip') }}"
  tasks:
    - debug:
        var: result

(details)

  1. If you can't or don't want to use community.general.lists_mergeby zip the lists and combine items. This gives also the desired result
result: "{{ list2.0|zip(link_list)|zip(pool_list)|map('flatten')|map('combine')|list }}"
Vladimir Botka
  • 58,131
  • 4
  • 32
  • 63
  • The first option only combines 2 list elements, and it does not combine them correctly (even if they are sorted). The second option does not work because there less items in List1 than there will be in List2. List1 will contain at max 3 elements, whereas List2 can contain upwards of 30 elements. So it does not map properly when creating the link and pool dictionaries, and leaves out IP's. – cilles Jul 12 '22 at 19:50
  • List1 contains the LTM device names (ltm1.mydomain.com), a link to the API GET results (mydomain.com/API_RESULTS_LINK), and the LTM pool name (pool_service1_side_a) that I want to perform actions against. List2 contains pool member information specific to each LTM pool. so there will always be more pool members than there are pools. In this situation I only included 2 pool members per pool, but in reality it can be upwards of 10 or more pool members per pool. – cilles Jul 12 '22 at 19:58
  • If I am trying to link the Host IP addresses to the LTM Pools/API Links, I will always be leaving out additional host IP's as there will always be more hosts than there are pools, as there are multiple hosts per pool – cilles Jul 12 '22 at 20:03
  • Yes, ``"the first option only combines 2 list elements"``. You provided nested lists: ``"msg": [ [ ... ] ]``. I thought maybe you want to loop more routers in the same play later. Both first elements, *`list1.0`* and *`list2.0`* are lists. I added links with details. – Vladimir Botka Jul 13 '22 at 03:32
0

With the help of a peer, I was able to figure it out by looping the second list, adding a selectattr filter to match my attribute against the first list, and combining that result with the second list to create a new list of results.

Filtered Loop:

- name: Combine lists
  set_fact:
    new_list: "{{ new_list | default([]) + [item|combine(filter_time)] }}"
  loop: "{{ list2 }}"
  vars:
    filter_time: "{{ list1 | selectattr('device', '==', item.device) | first }}"

Output:

ok: [localhost] => {
    "msg": [
        [
            {
                "device": "LTM1_Device",
                "host": "/Common/LTM1_Host1",
                "ip": "0.0.0.1",
                "link": "LTM1_Link",
                "ltm_pool": "LTM1_Pool",
                "port": "5555"
            },
            {
                "device": "LTM1_Device",
                "host": "/Common/LTM1_Host2",
                "ip": "0.0.0.2",
                "link": "LTM1_Link",
                "ltm_pool": "LTM1_Pool",
                "port": "5555"
            },
            {
                "device": "LTM2_Device",
                "host": "/Common/LTM2_Host1",
                "ip": "0.0.0.3",
                "link": "LTM2_Link",
                "ltm_pool": "LTM2_Pool",
                "port": "5555"
            },
            {
                "device": "LTM2_Device",
                "host": "/Common/LTM2_Host2",
                "ip": "0.0.0.4",
                "link": "LTM2_Link",
                "ltm_pool": "LTM2_Pool",
                "port": "5555"
            },
            {
                "device": "LTM3_Device",
                "host": "/Common/LTM3_Host1",
                "ip": "0.0.0.5",
                "link": "LTM3_Link",
                "ltm_pool": "LTM3_Pool",
                "port": "5555"
            },
            {
                "device": "LTM3_Device",
                "host": "/Common/LTM3_Host2",
                "ip": "0.0.0.6",
                "link": "LTM3_Link",
                "ltm_pool": "LTM3_Pool",
                "port": "5555"
            }
        ]
    ]
}
cilles
  • 21
  • 3