Massive text file to be read into R

Question

so the story is i have a 30 gig txt file that need to be read into R, it contains two cols and about 2 billion rows of integers! I dont want to load the whole thing in one go, sizeable chunks will suffice.

I've tried using read.table with arguments like nrow = 10000000 and skip = "stupidly_large_number"

but i get the following error when i get far through the file

Error in readLines(file, skip):
    cannot allocate vector of length 1800000000

Please help me get at the data and thanks in advance!

score 0 · Accepted Answer · answered Oct 18 '13 at 20:02

0

it seems to me that you may need to split the text file into manageable chunks first before trying to process them. The unix split command should do the trick, but I don't know if you're on a platform that command exists on.

answered Oct 18 '13 at 20:02

stonecrusher

1,246
10
12

would their be anything similar that i could do on a windows os? – David Oct 18 '13 at 20:19
2

Actually just install cygdrive and used the split command, cheers! – David Oct 18 '13 at 22:29

Massive text file to be read into R

1 Answers1