1

I'm trying to extract some data from the aws ec2 describe-security-groups output, looking for ingress rules which have a description that starts with a value (i.e. Temp-JDoe ...), using filters to restrict the results returned. I'm getting the result I want, but in a deeply nested array with an embedded array. I want to combine all values at the first level with the values from the nested level pulled up, joined with ":", so I can then iterate over the resulting multi-line string, creating an aws ec2 revoke-security-group-ingress command to delete these rules.

I thought I had painstakingly worked out every part of what's turning out to be a complex JMESPath query, where each part works independently, but when combined, I'm getting an error I can't figure out.

I also currently have two filters that seem to be necessary to eliminate first level array values when the matched set at the second array level is null, that seems convoluted, and am wondering if there's a more elegant way to obtain the same result.

Our goal is to create a pair of bash scripts which cloud admins can run to quickly add and remove ad-hoc security group rules to bastion host security groups, when they have to work from an unknown location. I think this is likely to be a problem others will have. On create, we want to first clear all rules which have a description "Temp-JDoe *", then create the needed set with descriptions like "Temp-JDoe (SSH)" containing the current IP, so when work is done, rules can again be found via this query and removed.

Most inspiration came from here: https://opensourceconnections.com/blog/2015/07/27/advanced-aws-cli-jmespath-query/

I found one similar question here: JMESPath - Joining items in a nested array, but it wasn't a good match due to my use of filters.

This statement returns the data I want. Showing json output format to show the structure I want to collapse and join:

prefix=Temp
username=JDoe
aws ec2 describe-security-groups --group-id $sg \
                                 --query 'SecurityGroups[].IpPermissions[?not_null(IpRanges[?Description!=`null`]|[?starts_with(Description, `'$prefix-$username'`) == `true`])].[IpProtocol,FromPort,ToPort,IpRanges[?Description!=`null`]|[?starts_with(Description, `'$prefix-$username'`) == `true`].[CidrIp,Description]]' \
                                 --profile $profile --region $region --output json

With this raw output:

[
    [
        [
            "tcp",
            22,
            22,
            [
                [
                    "77.111.222.223/32",
                    "Temp-JDoe (SSH)"
                ]
            ]
        ],
        [
            "udp",
            1194,
            1194,
            [
                [
                    "77.111.222.223/32",
                    "Temp-JDoe (VPN-UDP)"
                ]
            ]
        ]
    ]
]

Which I need to get into this final form:

udp:1194:1194:70.185.154.223/32:Temp-JDoe (VPN-UDP)
tcp:22:22:70.185.154.223/32:Temp-JDoe (SSH)

While this statement with joins works without filters to restrict array data (getting all at the outer level and the 1st at the inner level):

aws ec2 describe-security-groups --group-id $sg \
                                 --query 'SecurityGroups[].IpPermissions[].[join(`:`,[IpProtocol,to_string(FromPort),to_string(ToPort),IpRanges[0].join(`:`,[CidrIp,Description])])]' \
                                 --profile $profile --region $region --output text

Example output (wrong records, right format):

[
    [
        "tcp:22:22:111.55.111.123/32:Home-FName (SSH)"
    ],
    [
        "udp:1194:1194:111.55.111.123/32:Home-FName (VPN-UDP)"
    ],
    [
        "tcp:943:943:70.185.154.223/32:Home-JDoe (Console)"
    ],
    [
        "tcp:443:443:111.55.111.123/32:Home-FName (VPN-TCP)"
    ],
    [
        "icmp:-1:-1:192.66.55.0/24:Office-NewYork (ICMP)"
    ]
]

Once I combine the two working subsets to filter on the data I want, then join it, it doesn't work, some issue with the second join. Here's that combined statement:

aws ec2 describe-security-groups --group-id $sg \
                                 --query 'SecurityGroups[].IpPermissions[?not_null(IpRanges[?Description!=`null`]|[?starts_with(Description, `'$prefix-$username'`) == `true`])].[join(`:`,[IpProtocol,to_string(FromPort),to_string(ToPort),IpRanges[?Description!=`null`]|[?starts_with(Description, `'$prefix-$username'`) == `true`].join(`:`,[CidrIp,Description])])]' \
                                 --profile $profile --region $region --output text

And the error message:

In function join(), invalid type for value: ['70.185.154.223/32:Temp-JDoe (VPN-UDP)'], expected one of: ['array-string'], received: "list"
mjcconsulting
  • 75
  • 1
  • 8

1 Answers1

1

I figured this out on my own. In the nested IpRanges array, [0] returns the 1st element as a string, while the [?starts...] filter notation returns an array of strings, even if it's just one string. So, we can pipe to get the first element:

aws ec2 describe-security-groups --group-id $sg \
                                 --query 'SecurityGroups[].IpPermissions[?not_null(IpRanges[?Description!=`null`]|[?starts_with(Description, `'$prefix-$username'`) == `true`])].[join(`:`,[IpProtocol,to_string(FromPort),to_string(ToPort),IpRanges[?Description!=`null`]|[?starts_with(Description, `'$prefix-$username'`) == `true`]|[0].join(`:`,[CidrIp,Description])])]' \
                                 --profile $profile --region $region --output text

Note this does suffer from one drawback - it only returns the first of what may be more than one CIDR with a matching description. But, as this query is already complicated enough, and for my use-case one is normally the only result, I'm just looping in the script which calls this to check for additional rules after deleting what's found on the first iteration.

It would still be nice to have a more efficient simpler way to run the two filters, removing the first not_null, and piping a result with null IpRanges to a second query that looked for an pruned that. Perhaps another day...

mjcconsulting
  • 75
  • 1
  • 8