Recursively cat all the files into single file

Question

I have bunch of files sitting in folders like

data\A\A\A\json1.json
data\A\A\A\json2.json
data\A\A\B\json1.json
...
data\Z\Z\Z\json_x.json

I want to cat all the jsons into one single file?

The result won't be valid JSON, are you sure you want to do this? — Barmar, Jun 05 '14 at 20:01

score 124 · Accepted Answer · answered Jun 05 '14 at 20:01

124

find data/ -name '*.json' -exec cat {} \; > uber.json

a short explanation:

find <where> \
  -name <file_name_pattern> \
  -exec <run_cmd_on_every_hit> {} \; \
    > <where_to_store>

answered Jun 05 '14 at 20:01

Pavel

7,436
2
29
42

13

Without using find: `cat ./data/*/*/*/*.json > ./all.json` – Mr. Llama Jun 05 '14 at 20:28
1

Or in `bash` 4 with `shopt -s globstar`, `cat ./data/**/*.json`. – chepner Jun 05 '14 at 21:01
4

You should also add "-type f"(only files) to avoid errors when cat tries to print a directory. – Bojan Hrnkas Sep 20 '16 at 07:40
3

What is \; doing? – rwitzel Apr 12 '19 at 08:08
1

@rwitzel `;` ends the `exec` command. You need to escape it with `\` though otherwise it would end the entire command. More info: https://askubuntu.com/questions/339015/what-does-mean-in-a-linux-command – ZaydH Jul 03 '19 at 15:20

score 18 · Answer 2 · answered Jun 05 '14 at 20:04

18

Use find to get all the JSON files and concatenate them.

find data -name '*.json' -exec cat {} + > all.json

Note that this will not be valid JSON. If you want a JSON file to contain multiple objects, they need to be in a containing array or object, so you'd need to add [ ] around them and put , between each one.

answered Jun 05 '14 at 20:04

Barmar

741,623
53
500
612

3

Strictly true, but it would be quite perverse to have directories with `.json` extensions. – Barmar Apr 14 '15 at 08:39
How do you add the ',' chars in the command above between each file? – Gal Bracha Nov 09 '17 at 20:01
1

@GalBracha Use a `find` command that just prints the filenames, pipe that to a script that reads each filename, does `cat "$filename"; echo ','`. – Barmar Nov 09 '17 at 20:47
1

You'll then have to edit the file when done to remove the last `,` – Barmar Nov 09 '17 at 20:48
@rwitzel If you end the `-exec` option with `+` instead of `;` it replaces `{}` with all the filenames instead of running the command separately for each filename. – Barmar Apr 12 '19 at 17:58

score 11 · Answer 3 · answered Jun 05 '14 at 20:10

11

Alternatively -- if you have a list of your files -- you can pipe that to xargs

<path to your files> | xargs cat > all.json

answered Jun 05 '14 at 20:10

csiu

3,159
2
24
26

2

to get a list recursively: find \`pwd\` – Alan Dec 10 '15 at 15:09

score 9 · Answer 4 · edited Apr 21 '19 at 11:19

9

find ./ -type f | xargs cat > ../singlefilename

I would like this ,easy and simple.

../ avoid the error input file is output file.

edited Apr 21 '19 at 11:19

danius

2,664
27
33

answered Jul 29 '18 at 04:11

LuciferJack

781
11
12

`find ./ -type f | fgrep -v singlefilename | xargs cat > ../singlefilename` – coyot Jan 13 '21 at 18:06

Recursively cat all the files into single file

4 Answers4

Linked