Reading very large CSV-files with many column in R

Asked Sep 20 '18 at 13:57

Active Jan 11 '22 at 23:08

Viewed 1,393 times

I am dealing with very large csv-files of 1-10 GB. I have figured out that I need to use the ff-package for reading in the data. However, this does not seem to work. I suspect that the problem is that I have approximately 73 000 columns and since ff reads row-wise, the size is to high for R's memory. My computer has 128 GB of memory, so the hardware should not be a limitation.

This there any way of reading the data column-wise instead?

Note: In each file there is 10 rows of text which needs to be removed before the file can be read as a matrix successfully. I have previously dealt with this by using the read.csv(file,skip=10,header=T,fill=T) on smaller files of same type.

Here is a picture of how a smaller version of the data sets looks in excel:

edited Jan 11 '22 at 23:08

lovalery

4,524
3
14
28

asked Sep 20 '18 at 13:57

Maria KA

5

i swear by `data.table::fread`; very fast and highly configurable. also easy to drop/select columns while reading.. to decrease memory-use – Wimpel Sep 20 '18 at 14:01
Package colbycol reads columnwise. – G. Grothendieck Sep 20 '18 at 14:09
@Wimpel It worked! You have officially saved my sanity – Maria KA Sep 20 '18 at 14:39
1

@Tim No, I actually have 128 GB of RAM. I don't need it for statistical analysis, but I do need to be able to upload the data so I can reduce it before analysis :) – Maria KA Sep 20 '18 at 14:41
@MariaKA glad to help. – Wimpel Sep 20 '18 at 14:52

Reading very large CSV-files with many column in R

0 Answers0