1

i have a file "stats.json" with the following initial structure"

[{
"price": 66.55,
"created": "2018-02-24T14:32:57"
}]

With a bash/curl script i take every 10 minutes data from an api and save it to "temp.json"

{
"price": 88.55,
"created": "2018-02-24T15:32:57"
}

I would like to merge the temp.json (which is updated every 10min) and to fill the stats.json file. I tried to do it with "jq" but without success.

Chris Stryczynski
  • 30,145
  • 48
  • 175
  • 286
nelasx
  • 191
  • 1
  • 2
  • 9
  • 1
    This is going to get slower and more expensive the longer your file is (and moreover, constant rewriting of old data increases risk of corruption or loss); I *strongly* suggest taking peak's advice, and changing to a format you can append to as a constant-time operation. – Charles Duffy Feb 24 '18 at 17:15
  • thanks for the heads up, the data older than 7-14 days will be purged. – nelasx Feb 24 '18 at 18:10

1 Answers1

4

One way to add the item in temp.obj to the array in stats.json:

jq —-slurpfile temp temp.obj '. + $temp' stats.json

Another way:

jq —s '.[0] + [.[1]]' stats.json temp.json

(If you want to overwrite stats.json, then I’d suggest using sponge as illustrated below.)

However, if at all possible, it would almost certainly be better to maintain stats.json as a log file in JSONL format: http://jsonlines.org/

That is, each object in temp.json would be appended as a single line to stats.json:

jq -c temp.json >> stats.json

This of course assumes that stats.json is initially empty or already in the JSONL format. To convert your existing stats.json to JSONL, you could do something like this:

cp -p stats.json stats.json.bak

jq -c .[] stats.json | sponge stats.json
Inian
  • 80,270
  • 14
  • 142
  • 161
peak
  • 105,803
  • 17
  • 152
  • 177
  • jq -c temp.json >> stats.json Is there a way to add an comma after the input? – nelasx Feb 28 '18 at 11:50
  • Yes, of course, but why would you want to? That would break everything. m The idea is to use line breaks as the "delimiter". – peak Feb 28 '18 at 17:34
  • @peak I guess I do understand @nelasx; I guess we both failed to understand (him and I) is that a file full of objects will also work as a file that has [{object},{object}] but your post made me aware of this possibility! tks – Thiago Conrado Mar 05 '20 at 20:32