uncompress a big .gz file

Question

I need to uncompress a transactions.gz file downloaded from Kaggle; approximately (2.86 GB), 350 million rows, 11 columns.

I tried on RStudio, windows Vista, 32 bits, RAM: 3 GB:

transactions <- read.table(gzfile("E:/2014/Proyectos/Kaggle/transactions.gz"))
write.table(transactions, file="E:/2014/Proyectos/Kaggle/transactions.csv")

But i receive this error message on the console

> transactions <- read.table(gzfile("E:/2014/Proyectos/Kaggle/transactions.gz"))
Error: cannot allocate vector of size 64.0 Mb
> write.table(transactions, file="E:/2014/Proyectos/Kaggle/transactions.csv")
Error: cannot allocate vector of size 64.0 Mb

I checked this case, but it didn't work for me: Decompress gz file using R

I would appreciate any suggestions.

What platform are you on? If linux-based you can interface with the unix/shell terminal and process it there before you bring it into R. — Rich Scriven, May 01 '14 at 01:23

James King · Accepted Answer · 2014-05-01T02:57:13.503

4

This file decompresses to a 22GB .csv file. You can't process it all at once in R on your 6GB machine because R needs to read everything into memory. It would be best to process it in an RDBMS like postgresql. If you are intent on using R you could process it in chunks, reading a manageable number of rows at a time: read a chunk, process it, and then overwrite with the next chunk. For this data.table::fread would be better than the standard read.table.

Oh, and don't decompress in R, just run gunzip from the command line and then process the csv. If you're on Windows you can use winzip or 7zip.

edited May 01 '14 at 02:57

answered May 01 '14 at 01:19

James King

6,229
3
25
40

I don't like applications that delete their input files, and thus don't use `gunzip` directly. Rather, I'll use `gzip -c -d`. – Matthew Lundberg May 01 '14 at 02:59
OK, but this prints to stdout and requires redirection, right? `gzip -c -d file.csv.gz > file.csv`, or `zcat file.csv.gz > file.csv`. Probably `gunzip -k file.csv.gz` is simplest. – James King May 01 '14 at 03:20
Sure, good luck in the competition. I am in Shoppers as well. – James King May 01 '14 at 14:21
Nice, I am going to explore purchase behavior to predict. – user3591356 May 02 '14 at 17:20
I guess you are using amazon dataset, aha – Statham Nov 01 '18 at 03:40

uncompress a big .gz file

1 Answers1