4

I need to parse a big JSON file (about 200MB). With json_decode I get a memory error

I have found salsify/jsonstreamingparser library that promise parser JSON documents. But I only have found in-memory examples that produce the same error of json_decode.

Anyone knows how can I parse large JSON files?

gvd
  • 1,482
  • 6
  • 32
  • 58
  • Raise your memory limit before parsing and lower it right after parsing. There really isn't much else you can do unless you contrive a hack with parsing the file using a C library and then reading it into PHP. – MonkeyZeus Feb 13 '18 at 18:26
  • check this link you might get some help : https://stackoverflow.com/questions/15373529/parse-large-json-file – prakash tank Feb 13 '18 at 18:34
  • Increase memory limit is not an option. – gvd Feb 13 '18 at 18:34
  • @prakashtank I don't have clear how this lib is working https://github.com/kuma-giyomu/JSONParser – gvd Feb 13 '18 at 19:01
  • The streaming parser will only help if you want to extract or work on *parts* of the JSON data. It sounds like you want to decode the whole thing. If you don't have enough memory to even hold a single copy of the data there's no library on earth that can help you. – Sammitch Feb 13 '18 at 19:13
  • I want to decode the entire file, but in small piece of data. Parts by parts – gvd Feb 13 '18 at 19:16
  • 1
    Then look at something like the [SubsetConsumerListener](https://github.com/salsify/jsonstreamingparser/blob/master/src/Listener/SubsetConsumerListener.php) in the salsify package as a starting point for building what you want. – Sammitch Feb 13 '18 at 19:19
  • Not sure how to implement abstract protected function consume($data); method to get the data – gvd Feb 13 '18 at 20:48
  • does it provide pagination or limit? – prakash tank Feb 14 '18 at 04:01
  • No, At least I didn't find it. You need to consume data inside Listener. It should be great have pagination. – gvd Feb 14 '18 at 15:20
  • Possible duplicate of [Processing large JSON files in PHP](https://stackoverflow.com/questions/4049428/processing-large-json-files-in-php) – user3942918 Feb 16 '18 at 08:06

2 Answers2

8

If you want to iterate over it and do not demand to have the whole structure in memory at once, try https://github.com/halaxa/json-machine. It allows you to simply iterate over a JSON of any size/length just using foreach.

Filip Halaxa
  • 728
  • 6
  • 13
-5

Depending on what webserver or php-version you use. It is basicly possible to set a value for max-used-memory and max-used-script-execution-time.

See :

PHP.NET - Limiting Ressources

PHP.NET - Limiting Execution-Time

IMO PHP is not made for this kind of cpu-heavy operations, i would prefer to use it in the more common way - just use a mysql, postgres, ... So parse all your data in your favourite way in your database. Problem solved :)

... hope so. HF

r4r3devAut
  • 90
  • 10
  • I discarted this option, because the size of the file could be larger than my webserver memory allowed. – gvd Feb 13 '18 at 19:03