How to import large JSONL files to R

Asked Jul 16 '21 at 17:35

Active Jul 16 '21 at 17:35

Viewed 307 times

I am trying to read large (>10gb) JSONL files in R, but I am getting an error message when I use the fromJSON function:

library(rjson)

data = "D:/Data/2020-02-01_tweets.jsonl"

result <- fromJSON(file=data)

Error in paste(readLines(file, warn = FALSE), collapse = "") : result would exceed 2^31-1 bytes

I did some research and it turns out JSONL files are different from regular JSON ones in the sense that each entry in the JSONL is valid JSON. How should I go about importing these files? I have 32gb of ram so size shouldn't be an issue.

asked Jul 16 '21 at 17:35

Thierry

1

try using `library(jsonlite)`. See this answer for large jsonl import guidance: https://stackoverflow.com/a/62311093/2070999 – SEAnalyst Jul 16 '21 at 17:54
2

`jsonlite::stream_in(file("D:/Data/2020-02-01_tweets.jsonl"))` – r2evans Jul 16 '21 at 17:58
1

could someone post this information as an answer? – Ben Bolker Jul 16 '21 at 18:44

How to import large JSONL files to R

0 Answers0