0

I'm new to json files in R. I got some json files for STONKS are you can see scraped online. The werid thing is each individual json file can be read perfectly using fromJSON from the jsonlite library. I have taken individual files to check on jsonlint.com to validate, and the web confirmed they are valid json files.

I have tried to use the validate () from jsonlite on single files, but there the file failed to pass the validation.

However, when I want to use lapply to read all the files R gave me error on

Error in parse_con(txt, bigint_as_char) : 
  lexical error: invalid char in json text.
                                   No data available for the given

Here are my simple codes:

library(tidyr)
library(jsonlite)

ls <- list.files(pattern = "*.json")

Data.fromJson <- lapply(ls, fromJSON)

Sorry but I think the problem might be with the data so I dont want to just trim it and make dummy as it might mess things up. Instead I have uploaded a few json files to my google drive and here is the link:

https://drive.google.com/drive/folders/1zM4vj1TIseFKBSiNWe5yMY9BJPg-CIsv?usp=sharing

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
ML33M
  • 341
  • 2
  • 19
  • 1
    I cannot replicate that error with your example files. What does seem odd is your `ls <- list.files(pattern = "*.json")` line. Does that return the files you expect? The `pattern=` arguemtn expects a regular expression, not a grob. I would have expected something like `ls <- list.files(pattern = "\\.json$")` – MrFlick Oct 16 '20 at 00:54
  • @MrFlick hmmm strange. The ls<- list.files() seemed to work fine though, it returned the character vector containing all the file names, displaying correctly , eg "ADBE-10-Day.json"..... Do my code worked fine in your hand? – ML33M Oct 16 '20 at 01:02
  • 1
    I got no errors reading on the data you provided. Everything was imported as a list. Do you get this error when you try to import these three files alone? It will be very hard to help without a proper reproducible example. – MrFlick Oct 16 '20 at 01:05
  • @MrFlick I knew what went wrong and you are right my friend. Indeed I also got the parsing error using the 3 files but then I noticed one odd thing, the char vector is 4 instead of 3. Then with what you commented on "\\.json$". What happened is I named my R code as Read json xxxx, and my fromJSON is trying to read my R code file..... classic screw up – ML33M Oct 16 '20 at 01:16
  • @MrFlick Strange things happened. If i change the name of the code, the 3 fiels run just fine. but when I deploy to all the files, problem remains the same error. I used the validate() on single file, the error says "lexical error: invalid char in json text.\n YUM-30-Day.json\n (right here) ------^\n" – ML33M Oct 16 '20 at 02:24
  • Are you sure you have the working directory set properly? The file paths you pass should be relative to the current working directory. If `fromJSON` cannot file a file with the name passed, it will try to interpret the filename itself as a JSON string which sounds like what is causing this particular error message. – MrFlick Oct 16 '20 at 02:26
  • @MrFlick thank you for the patience for a noob. If I read just a single file, fromJSON is fine. If I try to use validate() on the single file, R tells me False and gave me the above error. I did set the working directory to the folder. Now If I ran the apply() of the fromJSON, I get "Error in parse_con(txt, bigint_as_char) : lexical error: invalid char in json text. No data available for the given (right here) ------^" – ML33M Oct 16 '20 at 02:34
  • Without a proper reproducible example there's nothing more I can really say. It's probably still a file path problem of some sort. Sorry I can't help further. – MrFlick Oct 16 '20 at 02:36
  • @MrFlick Yes I know this is frustrating. And I appreciate your help. I have updated the link to include all the data to the folder. Maybe this time it could give you the error. – ML33M Oct 16 '20 at 02:44
  • 1
    Look at the files by size. Some are very small for example "GII-20-Day.json" That file doesn't contain valid JSON data. It's unclear what you want to do with that file. – MrFlick Oct 16 '20 at 02:57
  • @MrFlick these data are given to me. Perhaps I should work on the ones with decent size and test if that works – ML33M Oct 16 '20 at 03:22
  • @MrFlick again, I have to thank you. that 3 files with the same ticker is a literal string with no ''". that's why jsonlite complains – ML33M Oct 16 '20 at 04:03
  • 1
    ML33M, two ideas. I get ofthen these kinds of problems when I load in Windows(i.e. Windows 1252 encoding) jsons encoded and captured in UTF-8. The way I handle this is to create a function with an if {} else {} so that if fromJSON() can not read the json, it saves the input path in a seaprate txt. When the process is over I jsut manually check the files logged in said txt. – Nicolás Velasquez Feb 19 '21 at 17:56

0 Answers0