Best way to format large JSON file? (~30 mb)

Question

I need to format a large JSON file for readability, but every resource I've found (mostly online) doesn't deal with data say, above 1-2 MB. I need to format about 30 MB. Is there any way to do this, or any way to code something to do this?

Do you simply need to read it? Or you want to format the output before sending it do clients? — Stefano Sanfilippo, Nov 09 '13 at 11:09

pstadler · Accepted Answer · 2023-03-01T10:35:34.307

135

With python >= 2.6 you can do the following:

For Mac/Linux users:

cat ugly.json | python -m json.tool > pretty.json

For Windows users (thanks to the comment from dnk.nitro):

type ugly.json | python -m json.tool > pretty.json

edited Mar 01 '23 at 10:35

answered Nov 09 '13 at 11:17

pstadler

1,993
1
14
24

3

After this processing, I can open and navigate through 12 MB JSON easily with Visual Studio. – Tomas Kubes Sep 05 '19 at 10:54
1

47MB file fully searchable in VSC after this. – MrfksIV Nov 25 '20 at 09:32
2

For windows users: `type ugly.json | python -mjson.tool > pretty.json` – dnk.nitro Apr 15 '21 at 17:54
2

The only thing that worked for me for 900 MB JSON file. It took a while though! – EM0 Apr 28 '21 at 14:41
1

Using a recent Python3 version you can simplify the command `python -mjson.tool input.json > formatted.json`. – Robert Oct 18 '21 at 14:55
thanks dnk.nitro .. it worked for a 180 MB file. – Deepa Dhanalakota Feb 23 '22 at 22:25
It looks like Python limits the size of the file to 2Gb. So if you have really large files, this solution won't work for you. – Alexis Wilke May 25 '22 at 19:09
2

I'm unsure why all the windows pythons commands don't have a space between -m and json. Only this worked for me: `type ugly.json | python -m json.tool > pretty.json` (powershell) – Shaun Oct 14 '22 at 08:29
I just used it on a .json file that contained Japanese text and it's converting the Japanese text to "\u304a\u3061\u3093\u3061\u3093" strings. It formats the .json correctly but seems to also transform the data so be careful. – BenjaminK May 24 '23 at 07:53

jmng · Answer 2 · 2019-04-10T08:38:04.320

52

jq can format or beautify a ~100MB JSON file in a few seconds:

jq '.' myLargeUnformattedFile.json > myLargeBeautifiedFile.json

The command above will beautify a single-line ~120MB file in ~10 seconds, and jq gives you a lot of json manipulation capabilities beyond simple formatting, see their tutorials.

edited Apr 10 '19 at 08:38

answered Jan 29 '19 at 11:13

jmng

2,479
1
25
38

1

It loads everything into RAM – nicolas2008 Jul 18 '20 at 00:45
Worked fine on a 100MB file, but not on a 900MB file. :( – EM0 Apr 28 '21 at 14:13
5

In windows, I had to use "." instead of '.' – VoteCoffee Apr 30 '21 at 20:23
3

This worked for me on a 172MB file, whereas Visual Studio Code just stared me straight in the face and said "I won't do that, it's too big." when I asked it to "format" the document. Thanks! – Jeremy Iglehart Aug 04 '21 at 23:46
3

If anyone copy-pastes large JSON and wants to skip having a file, one can run `get-clipboard | jq "." | set-clipboard` in PowerShell – Aaron B Nov 22 '21 at 23:09
1

another window example: type big_unformatted.json | c:\path_to_jq\jq-win64.exe > big_formatted.json – miha Mar 28 '22 at 16:16
1

Good tip, worked great here... using WIndows 11 64 bits! Thanks! You can have it installed with ease if you use Chocolatey. – GlauberMD May 17 '22 at 13:46

nicolas2008 · Answer 3 · 2020-07-18T13:08:11.250

jsonpps is the only one worked for me (https://github.com/bazaarvoice/jsonpps).
It doesn't load everything to RAM unlike jq, jsonpp and others that I tried.

Some useful tips regarding installation and usage:

Download url: https://repo1.maven.org/maven2/com/bazaarvoice/jsonpps/jsonpps/1.1/jsonpps-1.1.jar

Shortcut (for Windows):

Create file jsonpps.cmd in the same directory with the following content:
@echo off java -Xms64m -Xmx64m -jar %~dp0\jsonpps-1.1.jar %*

Shortcut usage examples:

Format stdin to stdout:
echo { "x": 1 } | jsonpps
Format stdin to file
echo { "x": 1 } | jsonpps -o output.json
Format file to file:
jsonpps input.json -o output.json

score 1 · Answer 4 · answered Oct 09 '22 at 15:14

Background-- I was trying to format a huge json file ~89mb on VS Code using the command (Alt+Shift+F) but the usuals, it crashed. I used jq to format my file and store it in another file.

A windows 11 use case is shown below.

step 1- download jq from the official site for your respective OS - https://stedolan.github.io/jq/

step 2- create a folder in the C drive named jq and paste the executable file that you downloaded into the folder. Rename the file as jq (Error1: beware the file is by default an exe file so do not save it as 'jq.exe' save it only as 'jq')

step 3- set your path variable to the URL of the executable file.

step 4- open your directory on cmd where the json file is stored and type the following command - jq . currentfilename.json > targetfilename.json

replace currentfilename with the file name that you want to format replace targetfilename with the final file name that you want your data formatted in

within seconds you should see your target file in the same directory in a formatted version which can now be opened on VS Code or any editor for that matter. Any error related to the recognizability of jq as a command can be traced back with high probability to Error 1.

jq jquery json data-preprocessing data-cleaning

score 1 · Answer 5 · answered Feb 22 '23 at 09:17

You can use Notepad++ (https://notepad-plus-plus.org/downloads/) for formatting large JSON files (tested in Windows).

Install Notepad++
Go to Plugins -> Plugins Admin -> Install the 'Json Viewer' plugin. The plugin source code is present in https://github.com/kapilratnani/JSON-Viewer
After plugin installation, go to Plugins -> JSON Viewer -> Format JSON.

This will format your JSON file

Best way to format large JSON file? (~30 mb)

5 Answers5

Linked