Parse and filter complex JSON Response in Python (the easy compute method)

Question

I have a list of dictionaries (basically JSON Response of an endpoint)

I would need to Parse this 16000 lines of json objects and fitler the documents/objects which match criteria that whose leaf element/field : statusInfo/status in not "UP" and of those filtered objects, just return "name" , "serviceUrl","status"

example :

"ADMIN-V1"  "http://aws-ec2.aws.com:4435"  "Warning"

I have been researching about JSONPath module , but there is no good documentation about it, and I could not find any easier way. Any guidance is highly appreciated.

here is a snippet from long 16000 lines of JSON response.

[
    {
        "id": "9c108ec5",
        "name": "USER-V2",
        "managementUrl": "http://aws-ec2.aws.com:5784/",
        "healthUrl": "http://aws-ec2.aws.com:5784/health",
        "serviceUrl": "http://aws-ec2.aws.com:5784/",
        "statusInfo": {
            "status": "UP",
            "timestamp": 1566663146681,
            "details": {
                "description": " Eureka Discovery Client",
                "status": "UP"
            }
        },
        "source": "discovery",
        "metadata": {},
        "info": {
            "component": "user",
            "description": "User REST Resource",
            "version": "2.2.1",
            "git": {
                "commit": {
                    "time": "07/27/2018 @ 15:06:55 CDT",
                    "id": "b2a1b37"
                },
                "branch": "refs/tags/v2.2.1"
            }
        }
    },
    {
        "id": "1a381f20",
        "name": "ADMIN-V1",
        "managementUrl": "http://aws-ec2.aws.com:4435/",
        "healthUrl": "http://aws-ec2.aws.com:4435/health",
        "serviceUrl": "http://aws-ec2.aws.com:4435/",
        "statusInfo": {
            "status": "Warning",
            "timestamp": 1566663146682,
            "details": {
                "description": "Spring Cloud Eureka Discovery Client",
                "status": "Warning"
            }
        },
        "source": "discovery",
        "metadata": {},
        "info": {
            "description": "Exchange Admin REST Resource",
            "api": {
                "version": "1.2.1",
                "name": "admin",
                "link": "https://app.swaggerhub.com/apis/AWSExchange/admin/1.2.1"
            },
            "implementation": "admin",
            "version": "1.1.0",
            "git": {
                "commit": {
                    "time": "01/04/2019 @ 15:36:48 UTC",
                    "id": "39d5551"
                },
                "branch": "refs/tags/v1.1.0"
            }
        }
    }
]

@oguzismail - please undelete your answer. It’s helpful for contrasting and benchmarking the two approaches. — peak, Aug 25 '19 at 11:17

score 1 · Answer 1 · answered Aug 25 '19 at 00:58

If your json file contains one big array, you'll want to stream that file in truncating out the array. Then use fromstream/1 to rebuild the objects and filtering them out as you go.

I don't have a representative file to test out the performance myself, but give this a try:

$ jq --stream -n 'fromstream(1|truncate_stream(inputs))
    | select(.statusInfo.status != "UP")
    | .name, .serviceUrl, .statusInfo.status
' input.json

Parse and filter complex JSON Response in Python (the easy compute method)

1 Answers1