JMESPath to generate an array of objects from an object of objects

Question

I have the following example json document (object of objects):

{
   "key1": { "key1a": [{"key1aa": "value1aa"}, {"key1ab": "value1ab"}],... },
   "key2": { "key2a": [{"key2aa": "value2aa"}, {"key2ab": "value2ab"}],... },
   ...
}

And I try to reshape it to an Array of that objects like

[
  { "key1": { "key1a": [{"key1aa": "value1aa"}, {"key1ab": "value1ab"}],... }},
  { "key2": { "key2a": [{"key2aa": "value2aa"}, {"key2ab": "value2ab"}],... }},
  ...
]

What I can use is JMESPath inside Ansible's jinja2 filter json_query

How is that possible?

Thanks

David

score 2 · Answer 1 · edited Jun 20 '20 at 09:12

Given my_dict the play

  vars:
    my_dict:
      key1: { "key1a": ["key1aa": "value1aa", "key1ab": "value1ab"]}
      key2: { "key2a": ["key2aa": "value2aa", "key2ab": "value2ab"]}
  tasks:
    - set_fact:
        my_list: "{{ my_list|default([]) + [{item.key: item.value}] }}"
      loop: "{{ my_dict|dict2items }}"
    - debug:
        msg: "{{ my_list|to_json }}"

gives

msg: '[{"key2": {"key2a": [{"key2aa": "value2aa"}, {"key2ab": "value2ab"}]}},
       {"key1": {"key1a": [{"key1aa": "value1aa"}, {"key1ab": "value1ab"}]}}]'

The output was printed with yaml callback (export ANSIBLE_STDOUT_CALLBACK=yaml). Then the output was split manually after the first item of the list.

json_query

It is also possible to create the list with jmespath i.e. json_query. For example, the task below gives the same result

- set_fact:
    my_list: "{{ my_dict|
                 dict2items|
                 json_query('[].[{key: key, value: value}]')|
                 map('items2dict')|
                 list }}"

to_nice_yaml

It's easier to see the data's structure when printed with the filter to_nice_yaml. For example

- debug:
    msg: "{{ my_list|to_nice_yaml }}"

gives

  msg: |-
    -   key2:
            key2a:
            -   key2aa: value2aa
            -   key2ab: value2ab
    -   key1:
            key1a:
            -   key1aa: value1aa
            -   key1ab: value1ab

filter_plugins

It's also possible to use custom Filter plugins. For example with the plugin

$ cat filter_plugins/dict_utils.py

def dict2list(d):
    l = []
    for i in d:
        h = {i:d[i]}
        l.append(h)
    return l

class FilterModule(object):

    def filters(self):
        return {
            'dict2list': dict2list
        }

the task below gives the same result

- set_fact:
    my_list: "{{ my_dict|dict2list }}"

For your convenience:

Zeitounator · Answer 2 · 2020-01-05T13:10:28.730

It is not possible with jmespath (i.e. json_query) alone unless you are ready to loose the top level key. In this case, you can simply apply the wildcard jmespath expression (i.e. *) to your object.

If you do not want to loose the top level keys and you still want to keep jmespath as part of the equation, the following example meets those requirements but needs some more filters.

Some details about the scenario:

Transform the initial hash to a list of {key: XXX, value: YYY} hashes with dict2items
Use json_query to transform that list of hashes (i.e. [{}, {}]) to a list of list of hashes (i.e. [[{}], [{}]])
Map the items2dict filter on each element of the result list
Finally the list filter will transform the temporary object from map to a usable list.

---
- name: Transform hash to list with jmespath and dict2items/items2dict
  hosts: localhost
  gather_facts: false

  vars:
    my_object: {
      "key1": { "key1a": ["key1aa": "value1aa", "key1ab": "value1ab"] },
      "key2": { "key2a": ["key2aa": "value2aa", "key2ab": "value2ab"] }
    }

  tasks:
    - name: Do transform with json_query
      debug:
        msg: >-
          {{
            my_object
            | dict2items
            | json_query('[].[{key: key, value: value}]')
            | map('items2dict')
            | list
          }}

Which gives:


PLAY [Transform hash to list with jmespath and dict2items/items2dict] ***************************************************************************************************************************************************************************************************

TASK [Do transform with json_query] *************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": [
        {
            "key1": {
                "key1a": [
                    {
                        "key1aa": "value1aa"
                    },
                    {
                        "key1ab": "value1ab"
                    }
                ]
            }
        },
        {
            "key2": {
                "key2a": [
                    {
                        "key2aa": "value2aa"
                    },
                    {
                        "key2ab": "value2ab"
                    }
                ]
            }
        }
    ]
}

PLAY RECAP **************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Thanks for your answer. But in that way, I will loose the keys 'key1','key2','keyX' which are important. — David, Jan 04 '20 at 13:14
Good point, I was a little fast. Meanwhile I'm affraid it makes this not feasible with jmespath at all. — Zeitounator, Jan 04 '20 at 13:19
It not feasible with jmespath.... **alone**. Please see my updated answer where it is still part of the equation (but uses some more filters). — Zeitounator, Jan 04 '20 at 14:02

score 1 · Answer 3 · answered Jan 04 '20 at 12:43

1

Note that your array in both examples is not correct JSON syntax (array elements should be inside an object).

So, given this example:

cat file
{
  "key1": { "key1a": [{"key1aa": "value1aa"}, {"key1ab": "value1ab"}] },
  "key2": { "key2a": [{"key2aa": "value2aa"}, {"key2ab": "value2ab"}] }
}

You can use jq:

<file jq 'to_entries|map({(.key):.value})'

answered Jan 04 '20 at 12:43

oliv

12,690
25
45

Note that it is not a correct json syntax for sure, but still valid yaml leading to the exact same data structure as the one you fixed in your answer. [yq](https://pypi.org/project/yq/) (based on jq) can interpret it without any problems – Zeitounator Jan 04 '20 at 13:02
A yeah, your right. My issue. My JSON here is not valid json syntax. I corrected it. – David Jan 04 '20 at 13:17
I try to prevent jq in my ansible because of additional dependencies. And my json object is respond from elastic roles API via the Ansible's uri module. They did not respond an arry. Don't know why they do it in spaces API but not in the roles API – David Jan 04 '20 at 13:41

JMESPath to generate an array of objects from an object of objects

3 Answers3