0

I am writing a script that loops through about 30,000 json files, reading them using jsonlite. However, I always encounter a fatal error like the one below about 5,000 files into my loop. I've written a similar script in python without any issue, but I cannot figure out why it won't work in R. I installed R on another machine and got the same error. Both machines are Windows 7. I am confident that this has nothing to do with the content of the json files, and the point at which the error is thrown changes every time, but it's always around 5000.

The loop I've been using for testing looks like this

t <- 0

for (filename in filenamelist){
   t <- t + 1
   print(t)
   json_data = fromJSON(filename )
}

The error looks like this: 

Error in parseJSON(txt):
   parse error: client cancelled parse via callback return value

Error during wrapup: lazy-load database 'C:/PROGRA~1/R/R-31~1.2/library/base/R/base.rbd' is corrupt
user2689931
  • 383
  • 1
  • 6
  • 15
  • Have you tried one of the other json-reading packages? Smells like a bug in jsonlite that might not happen in rjson or RJSONIO... – Spacedman Jan 23 '15 at 17:47
  • Also, `RJSONIO::fromJSON` accepts handler functions for the C routine. Depending on what else you're doing, it might also be much faster to parse with a handler. – Rich Scriven Jan 23 '15 at 17:49
  • It looks like it may be an issue with jsonlite. Both rjson and rjsonio don't throw an error. I'll just have to format output into a data frame myself. – user2689931 Jan 23 '15 at 17:57
  • I've just tried all three with 17576 (26 cubed) JSON files of the mtcars data set. All three packages work fine, although jsonlite takes twice as long as rjson, even when you wrap the rjson return into a data frame with `do.call(rbind,rjson::fromJSON(file=filename))`. jsonlite also seems to take up more memory. – Spacedman Jan 23 '15 at 18:20

0 Answers0