Flatten nested dictionary to key/value pairs with Ansible

Question

---
- hosts: localhost
  vars:
    mydict:
      key1: val1
      key2: val2
      key3:
        subkey1: subval1
        subkey2: subval2
  tasks:
    - debug:
        msg: "{{ TODO }}"

How would I make the above debug message print out all key/value pairs from the nested dictionary? Assume the depth is unknown. I would expect the output to be something like:

{
  "key1": "val1",
  "key2": "val2",
  "subkey1": "subval1"
  "subkey2": "subval2"
}

Vladimir Botka · Accepted Answer · 2021-06-26T20:44:54.417

Write a filter plugin and use pandas.json_normalize, e.g.

shell> cat filter_plugins/dict_normalize.py 
from pandas.io.json import json_normalize

def dict_normalize(d):
    df = json_normalize(d)
    l = [df.columns.values.tolist()] + df.values.tolist()
    return(l)

class FilterModule(object):
    def filters(self):
        return {
            'dict_normalize': dict_normalize,
        }

The filter returns lists of keys and values

    - set_fact:
        mlist: "{{ mydict|dict_normalize }}"

gives

  mlist:
  - - key1
    - key2
    - key3.subkey1
    - key3.subkey2
  - - val1
    - val2
    - subval1
    - subval2

Create a dictionary, e.g.

    - debug:
        msg: "{{ dict(mlist.0|zip(mlist.1)) }}"

gives

  msg:
    key1: val1
    key2: val2
    key3.subkey1: subval1
    key3.subkey2: subval2

If the subkeys are unique remove the path

    - debug:
        msg: "{{ dict(_keys|zip(mlist.1)) }}"
      vars:
        _regex: '^(.*)\.(.*)$'
        _replace: '\2'
        _keys: "{{ mlist.0|map('regex_replace', _regex, _replace)|list }}"

gives

  msg:
    key1: val1
    key2: val2
    subkey1: subval1
    subkey2: subval2

Notes

Install the package, e.g. python3-pandas
The filter might be extended to support all parameters of json_normalize
The greedy regex works also in nested dictionaries

Q: "Turn the key3.subkey1 to get the original dictionary."

A: Use json_query. For example, given the dictionary created in the first step

    - set_fact:
        mydict_flat: "{{ dict(mlist.0|zip(mlist.1)) }}"

  mydict_flat:
    key1: val1
    key2: val2
    key3.subkey1: subval1
    key3.subkey2: subval2

iterate the keys and retrieve the values from mydict

    - debug:
        msg: "{{ mydict|json_query(item) }}"
      loop: "{{ mydict_flat|list }}"

gives

  msg: val1
  msg: val2
  msg: subval1
  msg: subval2

I didn't think about the uniqueness of the keys until reading your answer. My final goal was to search through the values and modify certain ones in place based on some criteria and I'll need the full path to do that. Would you know of a quick way to turn the `key3.subkey1` back into something that can be used in the original dict? For example: `mydict['key3']['subkey1']` — Joshua Gilman, Jun 26 '21 at 20:16
Sure, I've updated the answer. But, I'm not sure about the use case. In particular, there is no difference between "mydict|json_query(item)" and "mydict_flat[item]". Open a new question with the [mre] if you have more issues. — Vladimir Botka, Jun 26 '21 at 20:49

Flatten nested dictionary to key/value pairs with Ansible

1 Answers1