AWS CLI returns JSON with control codes making JQ fail

Question

I've used jq many times to parse, pick values etc from JSON returned by AWS CLI, e.g. for ec2 describe-instances etc.

Now I'm using the dockerized version of AWS CLI v2 to get a list of CloudWatch log groups:

$ alias aws='docker run --rm -it -v ~/.aws:/root/.aws -e AWS_PROFILE -e AWS_REGION amazon/aws-cli'
$ aws logs describe-log-groups

{
    "logGroups": [
        {
            ...
        }
    ]
}

This looks like proper JSON; however when piping this to jq, I get:

parse error: Invalid numeric literal at line 1, column 2

If looking at the JSON returned from aws in a binary editor, or using jq:s inputs feature, I see that it contains a lot of control codes:

[
  "\u001b[?1h\u001b=\r{\u001b[m\r",
  "    \"logGroups\": [\u001b[m\r",
  "        {\u001b[m\r",
  "            \"logGroupName\": \"/aws/lambda/...
...

It seems to me that it's the fact that I'm using AWS CLI through docker that causes this, because when using the old fashioned AWS CLI v1 installed using pip it does not happen - but it could also be v2 vs v1 that is the key difference rather than using docker as the environment I guess (I never tried installing v2 natively).

These control codes, e.g. \u001b[m, look like ANSI codes to control formatting such as bold, colors etc. But AFAIK the AWS CLI does not use colored/ANSI output. Why are they included in the returned JSON? Is there a simple tool to strip them away so that I can continue using the dockerized AWS CLI v2 and pipe the output to jq? I found other answers using complex sed patterns and I thought to myself that there must be a simpler way to do this?

Edit: here's a minimal example that shows the control codes using xxd. I deliberately listed log groups with a mismatching filter to get an empty array:

$ aws logs describe-log-groups --log-group-name-prefix FOO > foo.txt
$ xxd foo.txt
00000000: 1b5b 3f31 681b 3d0d 7b1b 5b6d 0d0a 2020  .[?1h.=.{.[m..  
00000010: 2020 226c 6f67 4772 6f75 7073 223a 205b    "logGroups": [
00000020: 5d1b 5b6d 0d0a 7d1b 5b6d 0d0a 0d1b 5b4b  ].[m..}.[m....[K
00000030: 1b5b 3f31 6c1b 3e                        .[?1l.>
$ cat foo.txt | jq
parse error: Invalid numeric literal at line 1, column 2
                                                        $

Same thing displayed with xxd when using non-dockerized AWS CLI v1:

00000000: 7b0a 2020 2020 226c 6f67 4772 6f75 7073  {.    "logGroups
00000010: 223a 205b 5d0a 7d0a                      ": [].}.

Answer 1

The aws CLI is feeding its output to less because a) you've allocated a pseudo-tty with the -it flags, b) as far as the process is concerned, it's outputting directly to the tty and not to a pipe, and c) you haven't told it do anything else instead. The correct fix is to remove -it - it's only there for when you need to provide interactive input, and if you're piping the output to jq then you don't need or want interactive input. However, if you're trying to set up an alias or function to behave seamlessly in place of aws, you need to decide whether or not to pass -it based on whether you want interactive input. You could try this:

function daws() {
    local usetty
    if [ -t 1 ]
    then
        usetty=-it
    else
        usetty=
    fi
    docker run --rm $usetty -v ~/.aws:/root/.aws -e AWS_PROFILE -e AWS_REGION amazon/aws-cli "$@"
}

And in one line:

function daws() { local usetty; if [ -t 1 ]; then usetty=-it; else usetty=; fi; docker run --rm $usetty -v ~/.aws:/root/.aws -e AWS_PROFILE -e AWS_REGION amazon/aws-cli "$@"; }

Alternatively you can pass --no-cli-pager or set the environment variable AWS_PAGER to cat, but I tried both and while they didn't result in any errors, there was still some odd messy output where newlines weren't applying a carriage-return for part of the output.