0

I've been trying to work with disk frame to load up a file that's about 45 gbs. I have used the code below to convert the csv to a disk frame:

output_path = file.path(tempdir(), "tmp_cars.df")

disk <- csv_to_disk.frame("full-drivers.csv", outdir = output_path, overwrite = T, header = T)

However, I keep getting:

"Error: cannot allocate vector of size 369.8 MB"

or the same error with 739.5 MB.

I tried to load up a second, smaller csv file to see if it works and it did. This second file is the same file as the large one but just with the first 1000 rows. Every other program but Rstudio and chrome is closed as I'm looking for solutions online. My PC has16 GB of ram and I'm confused to why the size is so small. Running memory.limit() gives me 16344.

Is there a way I can load up my first file without getting the vector error?

Shazzzam
  • 1
  • 2
  • 1
    Maybe use a different package like `vroom`, which also reads data files "on-demand". It is included in the most recent version of `readr`: https://readr.tidyverse.org/news/ – Gregor de Cillia Jul 23 '21 at 16:47
  • I will look into it and see if that helps. Thanks for the suggestion! – Shazzzam Jul 23 '21 at 16:49

0 Answers0