r-unzip slows down when reading more than 50 000 csv's in zip file

Asked Apr 10 '18 at 14:04

Active Apr 10 '18 at 15:00

Viewed 351 times

I'm trying to unzip a 7zip file with the unzip function. The zip file contains around 155 000 csv files inside. When it starts unzipping the first 25 000 csv files are read within 5 minutes and then it starts to slow down and reads all files in almost an hour. Is this common behavior.

I know not providing a reproducible example makes everyones life more difficult but I cannot share the data with which I'm working. I'm just interested whether there is another way to extract files from zip with r, tweak the function or maybe call outside program or something?

Extracting with 7zip outside of R takes 20 mins tops, which is why I believe this to be an R related issue

edited Apr 10 '18 at 15:00

asked Apr 10 '18 at 14:04

adl

1,390
16
36

2

Hint: It's your file system, not R. – Dirk Eddelbuettel Apr 10 '18 at 14:05
1

Try doing it outside of R and see if anything changes. Are you on Windows? Try deleting the files (outside of R) as it unzips them so the directory isn't getting larger and larger. – Mark Setchell Apr 10 '18 at 14:35
Also, have a word with whoever is daft enough to give you such a thing! – Mark Setchell Apr 10 '18 at 14:41
I updated the post – adl Apr 10 '18 at 15:01
How about if you use `system("7z x ABC")` from inside R? Are you using Windows? – Mark Setchell Apr 10 '18 at 16:30
windows10, yes. What does ABC represent ? – adl Apr 10 '18 at 18:18
I used `ABC` to represent your zipfile - because I don't know its name. I was suggesting you use `system()` fron within R to execute the unzipping. – Mark Setchell Apr 10 '18 at 20:22

r-unzip slows down when reading more than 50 000 csv's in zip file

0 Answers0