Stack Overflow
Stack Overflow

Questions tagged [ijson]

Ijson is an iterative JSON parser with standard Python iterator interfaces.

Links

66 questions
1
vote
2 answers

Continue iteration to the next json record after encountering ijson.common.IncompleteJSONError

I have a large json file (about 11,600 records) and I am trying to parse it using ijson. However, the for loop breaks because of one faulty json record. Is there a way to continue the iteration by skipping that record and moving on using ijson or…
aditi19
  • 11
  • 1
1
vote
1 answer

reading large file with ijson and f.seek(0) taking too much time

I have 100 thousand of very large JSON files that I need to process on specific elements. To avoid memory overload I am using a python library called ijson which works fine when I am processing every object with preceding f.seek(0) to point file…
A l w a y s S u n n y
  • 36,497
  • 8
  • 60
  • 103
1
vote
1 answer

ijson kvitems unexpected behaviour

I'm using ijson to parse through large JSONs. I have this code, which should give me a dict of values corresponding to the relevant JSON fields: def parse_kvitems(kv_gen, key_list): results = {} for key in key_list: results[key] =…
Tim Kirkwood
  • 598
  • 2
  • 7
  • 18
1
vote
1 answer

Getting 'parse error: unallowed token...' while parsing a big JSON with ijson

I'm trying to parse and sift through a very big JSON file, containing tweet metadata of 9gb size. That's why I'm using ijson since this was the one most recommended by the community for such files. Still pretty new at it but I rigged up this…
rick458
  • 97
  • 6
1
vote
1 answer

Python ijson - nested parsing

I'm working with a web response of JSON that looks like this (simplified, and I can't change the format): [ { "type": "0","key1": 3, "key2": 5}, { "type": "1","key3": "a", "key4": "b"}, { "type": "2", "data": [] } ] I…
user3459685
  • 87
  • 2
  • 5
1
vote
1 answer

Load a large json file 3.7GB into dataframe and convert to csv file using ijson

I have a large json data file with 3.7gb. Iam going to load the json file to dataframe and delete unused columns than convert it to csv and load to sql. ram is 40gb My json file structure {"a":"Ho Chi Minh City,…
Dinh Truong Anh Phuong
  • 127
  • 10
1
vote
0 answers

Are there any way to read JSON files in pieces?

I am trying to read some JSON file from web and create a SQL database with the data. I am using ijson to read data as stream. But when the code fails I need to start over to retrieve data. Are there any way to continue reading JSON file from where…
PsyyduckK
  • 11
  • 2
1
vote
1 answer

How to make buckets from a big Json file in Python?

I have the following local Json file (around 90MB): For my data to be accessible, I want to create smaller JSON files that include exactly the same data but only 100 of the array entries in Readings.SensorData every time. So a file that includes…
Mohammad
  • 7,344
  • 15
  • 48
  • 76
1
vote
2 answers

C wrapper Python library Json Slicer is slower than pure Python ijson library

According to the official documentation (https://pypi.org/project/jsonslicer/), the basic configuration of Json Slicer yields 586.5K objects/sec, ijson with Python at the back-end yields 32.2K objects/sec, while ijson with C back-end…
Parikshit Chitnis
  • 11
  • 2
1
vote
1 answer

Python: ijson.parse(in_file) vs json.load(in_file)

I am trying to read a large JSON file (~ 2GB) in python. The following code works well on small files but doesn't work on large files because of MemoryError on the second line. in_file = open(sys.argv[1], 'r') posts = json.load(in_file) I looked…
Course Attendance
  • 11
  • 1
  • 2
1
vote
1 answer

extracting values from json file using ijson

I have a large JSON file which looks like this : {"details":{ "1000":[ ["10","Thursday","1","19.89"], ["12","Monday","3","20.90"], ... ] "1001":[ ["30","Sunday","11","80.22"], …
Anmol
  • 339
  • 1
  • 4
  • 17
0
votes
0 answers

How to only partially parse a JSON file into NetworkX graph?

I have a JSON with the following format: { "directed": false, "multigraph": false, "nodes": [ { "bad_val": { ... } "id": "node_id" ] } This JSON represents a NetworkX graph created using the node_link_data…
jackhart591
  • 41
  • 6
0
votes
1 answer

Reading a very large json file in python, pandas, ijson

I have to process a JSON file that is very large (86 GB). I have tried a few different methods of parsing the file, but none of them completed without running out of memory or crashing my computer, and they also didn't seem to have the outcome I…
Tyler Moore
  • 133
  • 1
  • 9
0
votes
0 answers

I am using ijson to read large json files, I am trying to paralellel processing, not working

I am trying to read json files of 30gb and I can do it by using ijson, but to speeed up the process I am trying to use multiprocessing. but I am unable to make it work, I can see the n workers ready but only one worker is taking all the load of the…
meuto
  • 11
  • 4
0
votes
1 answer

How to load whole large (100 GB) json to memory with python

I have json file, which size is 100 gb. It scheme looks like: json_f = {"main_index":{"0":3,"1":7},"lemmas":{"0":["test0", "test0"],"1":["test1","test1"]}} *"lemmas" elements contain large lists with words. Len of "lemmas" elements about 2kk. As a…
SollPicher
  • 11
  • 1
1
2
3 4 5