Using json_query in Ansible to sort through complex JSON

Question

I've run into a frustrating problem with Ansible and trying to parse a complex JSON object.

I need to find the physical volume, logical volume, disk name, and partition name for the devices that are mounted as the root file system. I already have the physical and logical volume names from the ansible_lvm fact.

During the default gathering_facts step Ansible also collects facts about system disks into a dictionary variable called ansible_devices:

{
    "ansible_devices": {
        "dm-0": {
            "holders": [],
            "host": "",
            "links": {
                "ids": [
                    "dm-name-ubuntu--vg-ubuntu--lv",
                    "dm-uuid-LVM-4s1K7fFeMXzoaijoeifajoiejoifjSi5DMhqUB2o9jvEJ5IOSIFE0902djfoiGhXaLL10sR"
                ],
                "labels": [],
                "masters": [],
                "uuids": [
                    "e9a5105b-9a0d-46ea-a0f39-a983ur9apdo98fr3"
                ]
            },
            "model": null,
            "partitions": {},
            "removable": "0",
            "rotational": "0",
            "sas_address": null,
            "sas_device_handle": null,
            "scheduler_mode": "",
            "sectors": "955801600",
            "sectorsize": "512",
            "size": "455.76 GB",
            "support_discard": "512",
            "vendor": null,
            "virtual": 1
        },
        "loop0": {
            "holders": [],
            "host": "",
            "links": {
                "ids": [],
                "labels": [],
                "masters": [],
                "uuids": []
            },
            "model": null,
            "partitions": {},
            "removable": "0",
            "rotational": "0",
            "sas_address": null,
            "sas_device_handle": null,
            "scheduler_mode": "mq-deadline",
            "sectors": "129936",
            "sectorsize": "512",
            "size": "63.45 MB",
            "support_discard": "4096",
            "vendor": null,
            "virtual": 1
        },
        "loop1": {
            "holders": [],
            "host": "",
            "links": {
                "ids": [],
                "labels": [],
                "masters": [],
                "uuids": []
            },
            "model": null,
            "partitions": {},
            "removable": "0",
            "rotational": "0",
            "sas_address": null,
            "sas_device_handle": null,
            "scheduler_mode": "mq-deadline",
            "sectors": "129944",
            "sectorsize": "512",
            "size": "63.45 MB",
            "support_discard": "4096",
            "vendor": null,
            "virtual": 1
        },
        "loop2": {
            "holders": [],
            "host": "",
            "links": {
                "ids": [],
                "labels": [],
                "masters": [],
                "uuids": []
            },
            "model": null,
            "partitions": {},
            "removable": "0",
            "rotational": "0",
            "sas_address": null,
            "sas_device_handle": null,
            "scheduler_mode": "mq-deadline",
            "sectors": "188072",
            "sectorsize": "512",
            "size": "91.83 MB",
            "support_discard": "4096",
            "vendor": null,
            "virtual": 1
        },
        "loop3": {
            "holders": [],
            "host": "",
            "links": {
                "ids": [],
                "labels": [],
                "masters": [],
                "uuids": []
            },
            "model": null,
            "partitions": {},
            "removable": "0",
            "rotational": "0",
            "sas_address": null,
            "sas_device_handle": null,
            "scheduler_mode": "mq-deadline",
            "sectors": "109072",
            "sectorsize": "512",
            "size": "53.26 MB",
            "support_discard": "4096",
            "vendor": null,
            "virtual": 1
        },
        "loop4": {
            "holders": [],
            "host": "",
            "links": {
                "ids": [],
                "labels": [],
                "masters": [],
                "uuids": []
            },
            "model": null,
            "partitions": {},
            "removable": "0",
            "rotational": "0",
            "sas_address": null,
            "sas_device_handle": null,
            "scheduler_mode": "mq-deadline",
            "sectors": "109072",
            "sectorsize": "512",
            "size": "53.26 MB",
            "support_discard": "4096",
            "vendor": null,
            "virtual": 1
        },
        "loop5": {
            "holders": [],
            "host": "",
            "links": {
                "ids": [],
                "labels": [],
                "masters": [],
                "uuids": []
            },
            "model": null,
            "partitions": {},
            "removable": "0",
            "rotational": "0",
            "sas_address": null,
            "sas_device_handle": null,
            "scheduler_mode": "mq-deadline",
            "sectors": "137712",
            "sectorsize": "512",
            "size": "67.24 MB",
            "support_discard": "4096",
            "vendor": null,
            "virtual": 1
        },
        "loop6": {
            "holders": [],
            "host": "",
            "links": {
                "ids": [],
                "labels": [],
                "masters": [],
                "uuids": []
            },
            "model": null,
            "partitions": {},
            "removable": "0",
            "rotational": "0",
            "sas_address": null,
            "sas_device_handle": null,
            "scheduler_mode": "mq-deadline",
            "sectors": "0",
            "sectorsize": "512",
            "size": "0.00 Bytes",
            "support_discard": "4096",
            "vendor": null,
            "virtual": 1
        },
        "loop7": {
            "holders": [],
            "host": "",
            "links": {
                "ids": [],
                "labels": [],
                "masters": [],
                "uuids": []
            },
            "model": null,
            "partitions": {},
            "removable": "0",
            "rotational": "1",
            "sas_address": null,
            "sas_device_handle": null,
            "scheduler_mode": "mq-deadline",
            "sectors": "0",
            "sectorsize": "512",
            "size": "0.00 Bytes",
            "support_discard": "0",
            "vendor": null,
            "virtual": 1
        },
        "nvme0n1": {
            "holders": [],
            "host": "Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983",
            "links": {
                "ids": [
                    "nvme-Samsung_SSD_970_EVO_Plus_500GB_S4EVNX0W303511J",
                    "nvme-eui.0025385331b1d473"
                ],
                "labels": [],
                "masters": [],
                "uuids": []
            },
            "model": "Samsung SSD 970 EVO Plus 500GB",
            "partitions": {
                "nvme0n1p1": {
                    "holders": [],
                    "links": {
                        "ids": [
                            "nvme-Samsung_SSD_970_EVO_Plus_500GB_S4EVNX0W303511J-part1",
                            "nvme-eui.0025385331b1d473-part1"
                        ],
                        "labels": [],
                        "masters": [],
                        "uuids": [
                            "F0A4-B4A1"
                        ]
                    },
                    "sectors": "1101824",
                    "sectorsize": 512,
                    "size": "538.00 MB",
                    "start": "2048",
                    "uuid": "F0A4-B4A1"
                },
                "nvme0n1p2": {
                    "holders": [],
                    "links": {
                        "ids": [
                            "nvme-Samsung_SSD_970_EVO_Plus_500GB_S4EVNX0W303511J-part2",
                            "nvme-eui.0025385331b1d473-part2"
                        ],
                        "labels": [],
                        "masters": [],
                        "uuids": [
                            "c8811628-4d6d-4402-bba2-c0ff74015361"
                        ]
                    },
                    "sectors": "1857536",
                    "sectorsize": 512,
                    "size": "907.00 MB",
                    "start": "1103872",
                    "uuid": "c8811628-4d6d-4402-bba2-c0ff74015361"
                },
                "nvme0n1p3": {
                    "holders": [
                        "ubuntu--vg-ubuntu--lv"
                    ],
                    "links": {
                        "ids": [
                            "lvm-pv-uuid-0FPssp-8lMn-Q3cM-nkiK-lXs5-1aCQ-G2bbLd",
                            "nvme-Samsung_SSD_970_EVO_Plus_500GB_S4EVNX0W303511J-part3",
                            "nvme-eui.0025385331b1d473-part3"
                        ],
                        "labels": [],
                        "masters": [
                            "dm-0"
                        ],
                        "uuids": []
                    },
                    "sectors": "973811727",
                    "sectorsize": 512,
                    "size": "464.35 GB",
                    "start": "2961408",
                    "uuid": null
                }
            },
            "removable": "0",
            "rotational": "0",
            "sas_address": null,
            "sas_device_handle": null,
            "scheduler_mode": "none",
            "sectors": "976773168",
            "sectorsize": "512",
            "serial": "S4EVNX0W303511J",
            "size": "465.76 GB",
            "support_discard": "512",
            "vendor": null,
            "virtual": 1
        }
    }
}

In that variable I can see the the disk and its partition that is "held" by LVM, but I can't for the life of me figure out how to filter that single dictionary item.

I want to pick out the single device object where the partitions value has holders==["ubuntu--vg-ubuntu--lv"], but can't figure out the proper syntax.

I've tried json_query, selectattr and a number of other filters in combination with dict2items but I think I'm missing something.

The documentation on json_query seems pretty sparse, and it doesn't match up perfectly with what I know from the regular jq utility.

Any help would be greatly appreciated!

Please don't link to external pastebins. Include the information necessary to understand your question in the question itself (use an excerpt when possible). — larsks, Jul 25 '23 at 14:08
Apologies, it was a particularly long JSON object and I wanted to include the full thing for reference. — Mike Staffa, Jul 25 '23 at 14:10

score 0 · Answer 1 · answered Jul 25 '23 at 14:39

The json_query filter doesn't use jq; it's uses the jmespath language. The documentation is pretty clear about that:

You must manually install the jmespath dependency on the Ansible controller before using this filter. This filter is built upon jmespath, and you can use the same syntax. For examples, see jmespath examples.

If you follow that link to the jmespath.org, you'll find extensive documentation and examples.

Having said that...I think jmespath is a poor match for the sort of data you're working with, because you have essentially arbitrary nesting levels and jmespath doesn't seem to have an equivalent to xpath's // operation.

Sometimes a simpler solution is to drop a custom filter in your filter_plugins directory...

def find_holder(v, holder):
    for devname, devdata in v.items():
        if holder in devdata["holders"]:
            return devname, devdata

        if "partitions" in devdata:
            devname, devdata = find_holder(devdata["partitions"], holder)
            if devname is not None:
                return devname, devdata

    return None, None


class FilterModule:
    def filters(self):
        return {
            "find_holder": find_holder,
        }

And then use that in your playbook:

- debug:
    msg: "{{ ansible_devices|find_holder('ubuntu--vg-ubuntu--lv') }}"

Thank you for your help. I pretty much got frustrated out of using this entirely. I'm now just using regular shell commands to extract the information, which is considerably less of a headache. I'd have preferred to use Ansible's built-in utilities, but oh well. — Mike Staffa, Jul 26 '23 at 08:46

Using json_query in Ansible to sort through complex JSON

1 Answers1