0

We have a directory /our_jsons that has the files:

file1.json

{"team": 1, "leagueId": 1, "name": "the ballers"}
{"team": 2, "leagueId": 1, "name": "the hoopers"}

file2.json

{"team": 3, "leagueId": 1, "name": "the gamerrs"}
{"team": 4, "leagueId": 1, "name": "the drivers"}

file3.json

{"team": 5, "leagueId": 1, "name": "the jumpers"}
{"team": 6, "leagueId": 1, "name": "the riserss"}

and we need to stack these into a single file output_file.json, that simply has all of the JSONs in our directory combined / stacked on top of one another:

output_file.json

{"team": 1, "leagueId": 1, "name": "the ballers"}
{"team": 2, "leagueId": 1, "name": "the hoopers"}
{"team": 3, "leagueId": 1, "name": "the gamerrs"}
{"team": 4, "leagueId": 1, "name": "the drivers"}
{"team": 5, "leagueId": 1, "name": "the jumpers"}
{"team": 6, "leagueId": 1, "name": "the riserss"}

Is this possible to do with a bash command in Mac / Linux? We're hoping this is easier than combining ordinary JSONs because these are NDJSONs and so the files truly simply need to just be stacked on top of one-another. Our full data is much larger (~10GB of data split over 100+ newline-delimited JSONs), and we're hoping to find a decently-performant (under 2-5 minutes) solution if possible. I just installed and am reading docs on jq currently, and will update if we come up with a solution.

EDIT:

It looks like jq . our_jsons/* > output_file.json concats the JSONs, however the output is not an ND JSON but rather an ordinary (and invalid) JSON file...

Canovice
  • 9,012
  • 22
  • 93
  • 211

1 Answers1

2

cat tmp/* | jq -c '.' > tmp/output_file.json appears to get the job done!

Canovice
  • 9,012
  • 22
  • 93
  • 211