Ansible: How to merge list of dictionary objects with the same key getting one dictionary element with a list of thouse different values grouped?

Question

I have been struggled with Ansible for a time and finally requesting for any help if possible,

Having a set of Nifi policies:

"nifi_raw_policies": [
    {
        "action": "read",
        "group": "8c052e6c-0184-1000-0000-000072a0bb44",
        "resource": "/data/process-groups/8b6b5c9f-0184-1000-57e3-f27fc56dd4aa"
    },
    {
        "action": "read",
        "group": "8c0536d8-0184-1000-0000-000018ba98a9",
        "resource": "/data/process-groups/8b6b5c9f-0184-1000-57e3-f27fc56dd4aa"
    }
]

I have the need to group them to obtain:

"nifi_grouped_policies": [
    {
        "policy": "read/data/process-groups/8b6b5c9f-0184-1000-57e3-f27fc56dd4aa",
        "groups": [ "8c052e6c-0184-1000-0000-000072a0bb44","8c0536d8-0184-1000-0000-000018ba98a9" ]
    }
]

But until now I'm still unable to group by "action"+"resource" key getting a list of related "group" in a new field "groups". I have managed to obtain only a list of dictionaries in the form { "action"+"resource" : "group1" }, ..., { "action"+"resource" : "groupN" }

Related ansible task code:

- name: Declare policies (simplest)
  set_fact:
    nifi_raw_policies: [
        {
            "action": "read",
            "group": "8c052e6c-0184-1000-0000-000072a0bb44",
            "resource": "/data/process-groups/8b6b5c9f-0184-1000-57e3-f27fc56dd4aa"
        },
        {
            "action": "read",
            "group": "8c0536d8-0184-1000-0000-000018ba98a9",
            "resource": "/data/process-groups/8b6b5c9f-0184-1000-57e3-f27fc56dd4aa"
        }
    ]

- name: Declare policies - debug
  debug: var=nifi_raw_policies

- name: Combine action and resource
  set_fact:
    nifi_policies: >-
      {{ nifi_policies|default([])
        | union([{
            item.action + item.resource : item.group
            }])
      }}
  with_items: "{{ nifi_raw_policies }}"

- name: Combine action and resource - debug
  debug: var=nifi_policies

After several tries using combine, union, groupby ansible filters without success.. any insigth would be highly appreciated.

Best regards,

Answer 1

Create the structure by Jinja. For example, given the mre simplified data

  raw:
    - action: read
      group: 1
      resource: data
    - action: read
      group: 2
      resource: data

declare the dictionary how to rename the keys

    keys:
      group: groups
      resource: policy

and declare the structure

  grouped: |
    {% for i in raw|groupby('action') %}
    {{ i.0 }}:
    {% for k in i.1|json_query('[].keys(@)|[]')|unique %}
    {% if k != 'action' %}
      {{ keys[k] }}: {{ i.1|map(attribute=k)|unique }}
    {% endif %}
    {% endfor %}
    {% endfor %}

gives

  grouped: |-
    read:
      groups: [1, 2]
      policy: ['data']

You can get the values only

  grouped_vals: "{{ grouped|from_yaml|dict2items|map(attribute='value')|list }}"

gives

  grouped_vals:
    - groups: [1, 2]
      policy: [data]

The first item from the list is what you're looking for. Or, you can get it from the dictionary

  nifi_grouped_policies: "{{ (grouped|from_yaml).read }}"

gives

  nifi_grouped_policies:
    groups: [1, 2]
    policy: [data]

---

Example of a complete playbook for testing

- hosts: localhost

  vars:

    raw:
      - action: read
        group: 1
        resource: data
      - action: read
        group: 2
        resource: data

    keys:
      group: groups
      resource: policy
    grouped: |
      {% for i in raw|groupby('action') %}
      {{ i.0 }}:
      {% for k in i.1|json_query('[].keys(@)|[]')|unique %}
      {% if k != 'action' %}
        {{ keys[k] }}: {{ i.1|map(attribute=k)|unique }}
      {% endif %}
      {% endfor %}
      {% endfor %}
    grouped_vals: "{{ grouped|from_yaml|dict2items|map(attribute='value')|list }}"
    nifi_grouped_policies: "{{ (grouped|from_yaml).read }}"

  tasks:

    - debug:
        var: grouped
    - debug:
        var: grouped_vals|to_yaml
    - debug:
        var: nifi_grouped_policies|to_yaml