Getting the error : "SystemError: Negative size passed to PyString_FromStringAndSize"

Question

I am trying to parse huge JSON file (around 20GB). Trying to read one line at a time (each line is a JSON object) and extract the required details.

Example:

JSON file data looks like the one shown below

{
    {a: [], b: [], c: [], d: [],e: []},
    {a: [], b: [], c: [], d: [],e: []},
    .....,
    {a: [], b: [], c: [], d: [],e: []}, 
}

Snippet to parse:

count = 0;
with open(fileName) as fp:
    try:
        for line in fp:
        data_local = json.loads(line)
        count = count + 1
        #access the data_local["a"]
    except:
        print "Error found" , count , len(data_local["a"])

Error Message (when "except block" not used):

Traceback (most recent call last):
File "./xyzFile", line 606, in <module>
for line in fp:
SystemError: Negative size passed to PyString_FromStringAndSize

Output (when "except" block" is used)

Error found  65 5392287

Found something similar on stack overflow but that didn't help. Tried to debug by catching the exception. It throws the error after reading 65th JSON objects (lines). Each JSON object is huge(in size and no of values)

Any lead on this would be appreciated.

Thanks

Count was for debugging purpose only , I need to store the json values — user3214392, Jan 24 '17 at 09:51
You can't store a huge value unless you have huge internal memory. Try to store it in flat file or in database. — Jay Parikh, Jan 24 '17 at 09:54
Memory isn't an issue . I am testing in a system with enough memory(RAM). In case of memory crunch it should have given the error "Memory Error" — user3214392, Jan 24 '17 at 09:57
if the storage is limited, then it will be sending any garbage values and hence you are getting some negative values. I guess. — Jay Parikh, Jan 24 '17 at 09:59

score 0 · Answer 1 · edited Oct 11 '22 at 06:15

0

Remove the try/except and try again, will probably show the correct error, I think this error happens when the return is too big for the catch.

After your for is not indented either.

try like this:

count = 0
with open(fileName) as fp:
    for line in fp:
        data_local = json.loads(line)
        count = count + 1
        #access the data_local["a"]

edited Oct 11 '22 at 06:15

Andrew Ryan

1,489
3
15
21

answered Oct 06 '22 at 19:49

Felipe Matos

1

Getting the error : "SystemError: Negative size passed to PyString_FromStringAndSize"

1 Answers1