JQ - Merge two arrays

Question

I'm looking for JQ query that allows me to merge 2 arrays variables (not files) and also take me to overwrite first array with newer value from second array. For example:

#!/bin/bash -e

firstArrayVariable='
    [
        {
            "Key": "A B",
            "Value": "1 2"
        },
        {
            "Key": "C D",
            "Value": "3 4"
        },
        {
            "Key": "E F",
            "Value": "5 6"
        },
        {
            "Key": "G H",
            "Value": "9 10"
        }
    ]
'

secondArrayVariable='
    [
        {
            "Key": "A B",
            "Value": "1 2"
        },
        {
            "Key": "C D",
            "Value": "3 4"
        },
        {
            "Key": "G H",
            "Value": "11 12"
        },
        {
            "Key": "J K",
            "Value": "15 16"
        }
    ]
'

jq \
    --compact-output \
    --raw-output \
    --arg jqSecondArrayVariable "${secondArrayVariable}" \
    '. + $jqSecondArrayVariable // empty' \
<<< "${firstArrayVariable}"

I could not get it to work and I got following error

jq: error (at :19): array ([{"Key":"A ...) and string ("\n [\n ...) cannot be added

What I expect the result of the merged array is

[
    {
        "Key": "A B",
        "Value": "1 2"
    },
    {
        "Key": "C D",
        "Value": "3 4"
    },
    {
        "Key": "E F",
        "Value": "5 6"
    },
    {
        "Key": "G H",
        "Value": "11 12"
    },
    {
        "Key": "J K",
        "Value": "15 16"
    }
]

any helps would be very appreciated!

UPDATED

I tried to use --argjson as @peak suggested, it concatenate array but it could not merge 2 arrays. Result I got now is an array with duplicated objects

RomanPerekhrest · Accepted Answer · 2018-05-15T16:43:34.580

11

Assuming that 2 input arrays are named as firstArr and secondArr respectively.

with group_by() (to group objects by the crucial key "Key") and map() functions:

jq --argjson arr1 "$firstArr" --argjson arr2 "$secondArr" -n \
'$arr1 + $arr2 | group_by(.Key) | map(.[-1])'

The output:

[
  {
    "Key": "A B",
    "Value": "1 2"
  },
  {
    "Key": "C D",
    "Value": "3 4"
  },
  {
    "Key": "E F",
    "Value": "5 6"
  },
  {
    "Key": "G H",
    "Value": "11 12"
  },
  {
    "Key": "J K",
    "Value": "15 16"
  }
]

Alternatively and in such particular case, a faster way, you could apply the following trick with unique_by() function:

jq --argjson arr1 "$firstArr" --argjson arr2 "$secondArr" -n '$arr2 + $arr1 | unique_by(.Key)'

edited May 15 '18 at 16:43

answered May 15 '18 at 16:32

RomanPerekhrest

88,541
4
65
105

@NamNguyen, welcome, Also, try my second approach (as it's shorter) – RomanPerekhrest May 15 '18 at 16:38
yes, the second approach works PERFECTLY too and even better and shorter. Thanks so much – Nam Nguyen May 15 '18 at 16:39
What is the purpose of `-n` option? The man page says "Don't read any input at all! Instead, the filter is run once using null as the input. This is useful when using jq as a simple calculator or to construct JSON data from scratch." But I don't quite understand. – panc Jun 11 '21 at 23:44
@panc, That means we don't read from standard input or input file in such case, but instead working with global, shell variables (mentioned 2 arrays variables) – RomanPerekhrest Jun 12 '21 at 07:32

score 1 · Answer 2 · edited Dec 19 '20 at 20:12

1

You should be using --argjson; --arg interprets its argument as a JSON string. You will also have to modify your jq filter, because simply adding the arrays will result in their concatenation.

For further guidance, see e.g. Combining JSON by common key-value pairs

edited Dec 19 '20 at 20:12

Ouroborus

16,237
4
39
62

answered May 15 '18 at 16:27

peak

105,803
17
152
177

1

Thanks @peak , I tried `--argjson` , it concatenates 2 arrays but it could not merge them. After my changes, result comes out is a long duplicate objects array `[{"Key":"A B","Value":"1 2"},{"Key":"C D","Value":"3 4"},{"Key":"E F","Value":"5 6"},{"Key":"G H","Value":"9 10"},{"Key":"A B","Value":"1 2"},{"Key":"C D","Value":"3 4"},{"Key":"G H","Value":"11 12"},{"Key":"J K","Value":"15 16"}]` – Nam Nguyen May 15 '18 at 16:31

JQ - Merge two arrays

2 Answers2

Linked