jq json parser hash the field value

Question

I have a JSON file as the below:

[{
  "macAddress": "ac:5f:3e:87:d7:1a",
  "ip": "1.2.3.4"
},
{
  "macAddress": "ac:5f:3e:87:d7:2a",
  "ip": "1.2.3.4"
},
{
  "macAddress": "ac:5f:3e:87:d7:3a",
  "ip": "1.2.3.4"
}]

use jq to hash the macAddress field like so:

jq .[] | hash(.macAddress)

Can I define my own hash function and let jq to run the hash during the parsing process?

My expected hash function can be simple as using native linux command md5sum

echo -n "my_salt""42:12:20:2e:2b:ca" | md5sum
d973ea7c353e78ba1724efbc8054dfdc -

So the output json will be

[{
  "macAddress": "d973ea7c353e78ba1724efbc8054dfdc",
  "ip": "1.2.3.4"
},
{
  "macAddress": "d973ea7c353e78ba1724efbc8054d2er",
  "ip": "1.2.3.4"
},
{
  "macAddress": "d973ea7c353e78ba2324efbc8054d123",
  "ip": "1.2.3.4"
}]

What would you want the output to be: a) a list of hashed mac addresses, separated by line feeds, or b) a JSON object like the input just with the mac addresses replaced with their hashed values, or c) something else? Please provide sample output in the question. — Hans Z., Aug 03 '16 at 11:13
@HansZ. I have updated the requirements as per your request. — Xianlin, Aug 04 '16 at 02:29

score 1 · Accepted Answer · edited Feb 01 '18 at 05:42

My expected hash function can be simple as using native linux command md5sum

This stays native and might be suitable;

Invocation :

jq -c .[] "$jsonfile" |
while read -r jsonline ; do
  hashmac="$(jq --arg mysalt "$mysalt" -s -j '
        .[] | "\($mysalt)" + .macAddress' <<<"$jsonline" |
      md5sum | cut -d ' ' -f1)"
  jq --arg hashmac "$hashmac" -s -r '
      .[] | .macAddress |= "\($hashmac)"' <<<"$jsonline"
done

Example file - /tmp/testfile:

[{
  "macAddress": "ac:5f:3e:87:d7:1a",
  "ip": "1.2.3.4"
},
{
  "macAddress": "ac:5f:3e:87:d7:2a",
  "ip": "1.2.3.4"
},
{
  "macAddress": "ac:5f:3e:87:d7:3a",
  "ip": "1.2.3.4"
},
{
 "macAddress": "42:12:20:2e:2b:ca",
 "ip": "1.2.3.4"
}]

Result Output:

{
  "macAddress": "1f960fe4d24684ca44e5e67b6259362c",
  "ip": "1.2.3.4"
}
{
  "macAddress": "3527422754ecbfdd01d48b17fce87842",
  "ip": "1.2.3.4"
}
{
  "macAddress": "9bc8da72324448c3032a20fb67a31466",
  "ip": "1.2.3.4"
}
{
  "macAddress": "d973ea7c353e78ba1724efbc8054dfdc",
  "ip": "1.2.3.4"
}

Comments:

-j causes jq to not output a newline, equivalent to your echo -n example
Variables in this example are sent to jq as strings using -arg, and referenced as "\($var)" as opposed to escaping the variable directly, for example:
- "\($mysalt)" + .macAddress' (jq variable)
  
  Instead of:
- "'"$mysalt"'" + .macAddress' (direct shell substitution)
This example uses cut -d ' ' -f1 to trim off the -, but there's probably a better way

Alternate:

jq --arg hashmac "$hashmac" -s -r '.[] |= . + {"hashAddress":"\($hashmac)"}'

Would append the json

[
  {
    "macAddress": "ac:5f:3e:87:d7:1a",
    "ip": "1.2.3.4",
    "hashAddress": "1f960fe4d24684ca44e5e67b6259362c"
  }
]

etc.

score 0 · Answer 2 · answered Feb 01 '18 at 15:34

0

The accepted answer entails 2n + 1 invocations of jq, where n is the number invocations of the message digest function.

To reduce the number of invocations to just 2, see shell-out value to md5 (crypto) function

answered Feb 01 '18 at 15:34

peak

105,803
17
152
177