How can you read a gzipped parquet file in Python

Question

I need to open a gzipped file, that has a parquet file inside with some data. I am having so much trouble trying to print/read what is inside the file. I tried the following:

with gzip.open("myFile.parquet.gzip", "rb") as f:
    data = f.read()

This does not seem to work, as I get an error that my file id not a gz file. Thanks!

Two steps: 1) unzip it 2) use a lib that knows how to read parquet files like https://pypi.org/project/parquet/ — balderman, Aug 12 '20 at 07:56

Gabio · Accepted Answer · 2020-08-12T08:12:02.873

11

You can use read_parquet function from pandas module:

Install pandas and pyarrow:

pip install pandas pyarrow

use read_parquet which returns DataFrame:

data = read_parquet("myFile.parquet.gzip")
print(data.count()) # example of operation on the returned DataFrame

edited Aug 12 '20 at 08:12

answered Aug 12 '20 at 08:00

Gabio

9,126
3
12
32

How can you read a gzipped parquet file in Python

1 Answers1

Linked