1

I have a large JSON file which looks like this :

{"details":{

    "1000":[
        ["10","Thursday","1","19.89"],

        ["12","Monday","3","20.90"],
        ...
    ]
    "1001":[
        ["30","Sunday","11","80.22"],

        ["88","Wednesday","22","8.29"],
        ...
    ]

    }
}

Now I'm extracting the lists present in variables like "1000", "1001" from the "details" value using ijson (Interactive Json) using code given below :

import ijson as ijson
filename='Clean_Details.json'

with open(filename,'r') as f:
    objects=ijson.items(f,'details.1001.item')
    for row in objects:
        print(row)
    print("Done")

But the Problem is that : for the loop is not terminating in the above code. After printing the final list in 1001 it keeps running.

I'm guessing that the the Generator(objects) in above code is not encountering the StopIteration don't know why.

Could Anyone help? A little help would be appreciated.

Anmol
  • 339
  • 1
  • 4
  • 17
  • Do you have a real case file for testing? Using the sample file i couldn't reproduce the error. – Kaniabi Aug 18 '16 at 19:59

1 Answers1

2

Ok as it turns out because of large size of JSON file which is > 800MB with about more than a million records the parsing takes time to complete so it

The loop terminates eventually but takes some time to complete. On a pc with normal specs it definitely takes some time.

Also using :

import ijson as ijson

is way slower on very large files because most of the parsing takes place using python backend code so inorder to improve the speed,

It is way better to use

import ijson.backends.yajl2_cffi as ijson

because it has a backend in C language using cffi which does improves running time for the above code.

Anmol
  • 339
  • 1
  • 4
  • 17
  • Apparently some issue with the "cffi" module? This is the error message I received. File "/usr/local/lib64/python3.6/site-packages/ijson/backends/yajl2_cffi.py", line 5, in from cffi import FFI ModuleNotFoundError: No module named 'cffi' – MMEL May 26 '22 at 00:07