Is there a way to load local csv.gz file in huggingface dataloader?

Question

I'm using datasets library by huggingface to load csv dataset stored locally. The problem is, the dataset is compressed and is stored as a csv.gz file. Therefore, I'm not able to load it using load_dataset('csv', '<local_file_path>.csv') method in huggingface.

score 0 · Answer 1 · answered Apr 02 '22 at 15:22

0

Just decompress it with gunzip:

gunzip -v yourdataset.csv.gz

answered Apr 02 '22 at 15:22

cronoik

15,434
3
40
78

So, that means NO? – Mahavir I Apr 03 '22 at 15:25
@MahavirI: At the moment the answer is no. Loading a `csv.gz` leads to an error on my machine (the hard disk is filled with GB of cache files even if the file is very small. Smells like a bug.). – cronoik Apr 03 '22 at 16:25

Is there a way to load local csv.gz file in huggingface dataloader?

1 Answers1