2

What is an OS-agnostic solution to use fread on a zip? I can't seem to find one.

Setting up the stage

Let's create two dataframes, write them to disk, and put them in a zip archive (I stole this from: How to zip multiple CSV files in R?)

library(zip)
df1 <- head(mtcars)
df2 <- head(iris)
write.csv(df1, 'df1.csv')
write.csv(df2, 'df2.csv')
zip(zipfile='df.zip', files=list.files(path = getwd(), pattern = ".csv$"))

Now I want to read this zip into R

Let's say I want to read df1.csv from the zip.

fread('df.zip/df1.csv')

Error in fread("df.zip/df1.csv") : File 'df.zip/df1.csv' does not exist or is non-readable

I tried this from fread() of file from archive

fread('unzip -p df.zip/df1.csv')

Null data.table (0 rows and 0 cols)
Warning message:
In fread("unzip -p df.zip/df1.csv") : File '/var/folders/w5/kqy78qb17v176195dtyyc4pj40000gn/T//RtmpIlNSk8/filee1693cc7f89' has size 0. Returning a NULL data.table.

I do not understand what it is trying to import, but clearly not my dataframe of interest.

Can you help?

Edit 1

Unzipping first is not really an option. In practice, I am working with batches of highly compressible files. Usually ~ 3000 xls files, each 1M rows. 100 Gb unzipped / 8 Gb zipped. Needless to say it would be much more comfortable to read directly from the zip!

francoiskroll
  • 1,026
  • 3
  • 13
  • 24
  • 2
    does a solution like this using `vroom` work for you: https://community.rstudio.com/t/vroom-with-multiple-files-different-number-of-columns/52752 – brian avery Feb 12 '21 at 15:34
  • or perhaps if you unzip the directory first then try `fread_folder` from `easycsv`? https://rdrr.io/cran/easycsv/man/fread_folder.html – brian avery Feb 12 '21 at 16:59
  • 1
    Thank you. I'll specify in the question. I am working with batches of highly compressible files. It is ~ 3000 xls files, each 1M rows. Something like 100 Gb unzipped / 8 Gb zipped. Needless to say it would be much more comfortable to read from the zip directly. I'll try `vroom`, but I imagine it can't beat `fread` for speed? – francoiskroll Feb 12 '21 at 18:02
  • 1
    as far as the outcome goes, do you want all data from the 3000 files in one dataframe, or 3000 separate dataframes one from each file? – brian avery Feb 12 '21 at 19:31
  • I am doing more processing to only import certain rows/columns that match some conditions, but essentially one dataframe for each file. Eventually what I will do is loop through the names of the files I want to import. The only thing blocking me is to know how to read a single file from within a zip archive (selecting it by name), the rest will be okay. – francoiskroll Feb 16 '21 at 15:24
  • Instead of fread you could use library(archive) and library(readr) and read_delim(archive_read("XXX.zip", file=XX)) – Tom Wenseleers Aug 17 '22 at 22:00

1 Answers1

4

Having unzip installed, this solution works on my computer :

fread(cmd = 'unzip -p df.zip df1.csv')
                  V1  mpg cyl disp  hp drat    wt  qsec vs am gear carb
1:         Mazda RX4 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
2:     Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
3:        Datsun 710 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
4:    Hornet 4 Drive 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
5: Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
6:           Valiant 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1
Waldi
  • 39,242
  • 6
  • 30
  • 78
  • 1
    @francoiskroll, did it work with `unzip` + `fread`? – Waldi Feb 16 '21 at 07:31
  • Thanks! Having to call things outside of R is what I was trying to avoid, hence the 'OS-agnostic'. But if that's the only choice that's okay, still better than unzipping my whole folder. I can't find instructions for installing `unzip` on Mac though? Can you point me in the right direction? – francoiskroll Feb 16 '21 at 15:31
  • I don't use a Mac, but it [seems available](https://swissmacuser.ch/macos-terminal-bash-unzip-zip-file-with-tar/) – Waldi Feb 16 '21 at 15:47
  • Or `brew install unzip` : https://apple.stackexchange.com/a/149088 – Waldi Feb 16 '21 at 16:05