Replace \r from json in shell

Question

I have below JSON in a variable name TASK_DEFINTIION It has a \r character at the end of "image": "700707367057.dkr.ecr.us-east-1.amazonaws.com/php-demo:feature-feature01\r" under containerDefinitions

I am using TASK_DEFINITION_AFTER= 'echo $TASK_DEFINTIION | sed "s/\\r//g"' to remove the \r but seems it is removing all the hidden carriage return but not removing the one which is visible as regular character.

Any help would be highly appriciated.

{
      "memory": "1024",
      "networkMode": "awsvpc",
      "family": "ecs-php-demo",
      "placementConstraints": [],
      "cpu": "512",
      "executionRoleArn": "arn:aws:iam::700707367057:role/ecsTaskExecutionRole",
      "volumes": [],
      "requiresCompatibilities": [
        "FARGATE"
      ],
      "taskRoleArn": "arn:aws:iam::700707367057:role/ecsTaskExecutionRole",
      "containerDefinitions": [
        {
          "memoryReservation": 256,
          "environment": [],
          "name": "ecs-php-demo",
          "mountPoints": [],
          "image": "700707367057.dkr.ecr.us-east-1.amazonaws.com/php-demo:feature-feature01\r",
          "cpu": 0,
          "portMappings": [
            {
              "protocol": "tcp",
              "containerPort": 8080,
              "hostPort": 8080
            }
          ],
          "logConfiguration": {
            "logDriver": "awslogs",
            "options": {
              "awslogs-region": "us-east-1",
              "awslogs-stream-prefix": "ecs",
              "awslogs-group": "/ecs/ecs-php-demo"
            }
          },
          "essential": true,
          "volumesFrom": []
        }
      ]
    }

What produced the file? You should probably fix that so that it doesn't add the `\r` in the first place. — chepner, May 29 '20 at 13:19

Léa Gris · Answer 1 · 2020-05-29T15:56:28.553

2

Using jq rtrimstr to stay conform with the JSON syntax:

#!/usr/bin/bash

TASK_DEFINTIION="$(
  jq '.containerDefinitions[].image|=rtrimstr("\r")' <<<"$TASK_DEFINTIION"
)"

echo "$TASK_DEFINTIION"

man jq:

rtrimstr(str) Outputs its input with the given suffix string removed, if it ends with it.
      jq ´[.[]|rtrimstr("foo")]´
         ["fo", "foo", "barfoo", "foobar", "foob"]
      => ["fo","","bar","foobar","foob"]

edited May 29 '20 at 15:56

answered May 29 '20 at 12:12

Léa Gris

17,497
4
32
41

Why not just `TASK_DEFINITION=$(jq ... <<< "$TASK_DEFINITION")`? – chepner May 29 '20 at 13:21
1

You could also use `rtrimstr("\r")` instead of `gsub`. (I'm not sure I even understand how `gsub` is working: it clearly *does*, I just don't know *why*, given my understanding of `gsub`.) – chepner May 29 '20 at 13:31
@chepner I don't use the `=$(command)` assignment because it strips the trailing newline. – Léa Gris May 29 '20 at 14:19
2

The JSON value won't end with any significant newlines. – chepner May 29 '20 at 14:20
@chepner I updated with your `rtrimstr` suggestion. – Léa Gris May 29 '20 at 15:55

Philippe · Accepted Answer · 2020-05-29T16:36:50.627

1

In your command, first \ escapes the second \, so sed sees only one \

You need :

TASK_DEFINITION_AFTER="$(echo $TASK_DEFINTIION | sed "s/\\\\r//g")"

edited May 29 '20 at 16:36

answered May 29 '20 at 11:30

Philippe

20,025
2
23
32

do you think its a good idea to parse json with `sed` or `awk` ?, the answer has to be changed with `jq` – Mahmoud Odeh May 29 '20 at 14:07

Mahmoud Odeh · Answer 3 · 2020-05-29T14:56:19.317

you can use jq to replace key value:

jq '.containerDefinitions[].image="700707367057.dkr.ecr.us-east-1.amazonaws.com/php-demo:feature-feature01"' file.json

but unfortunately, jq does not support in-place editing, so you must redirect to a temporary file first and then replace your original file with it, or use sponge utility from the more utils package, like that:

jq '.containerDefinitions[].image="700707367057.dkr.ecr.us-east-1.amazonaws.com/php-demo:feature-feature01"' file.json|sponge file.json

A pure jq solution to remove \r with gsub :

jq '.containerDefinitions[].image|=gsub("[\r]"; "")' file.json|sponge file.json

output sample :

{
  "memory": "1024",
  "networkMode": "awsvpc",
  "family": "ecs-php-demo",
  "placementConstraints": [],
  "cpu": "512",
  "executionRoleArn": "arn:aws:iam::700707367057:role/ecsTaskExecutionRole",
  "volumes": [],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "taskRoleArn": "arn:aws:iam::700707367057:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "memoryReservation": 256,
      "environment": [],
      "name": "ecs-php-demo",
      "mountPoints": [],
      "image": "700707367057.dkr.ecr.us-east-1.amazonaws.com/php-demo:feature-feature01",
      "cpu": 0,
      "portMappings": [
        {
          "protocol": "tcp",
          "containerPort": 8080,
          "hostPort": 8080
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs",
          "awslogs-group": "/ecs/ecs-php-demo"
        }
      },
      "essential": true,
      "volumesFrom": []
    }
  ]
}

Replace \r from json in shell

3 Answers3