3

I plan to change data file format from parquet to feather. Parquet has compression options(lz4, etc) and I have used them. But I can not find them in feather or Arrow file. Is compression not supported?

makepossible99
  • 47
  • 5

1 Answers1

3

The Arrow file format, which is designed for memory-resident-data, does not support compression algorithms such as lz4, snappy, gzip, and zlib. The only compression currently supported by Arrow is dictionary compression, a scheme that usually does not require decompression before data processing.

Ajay Kharade
  • 1,469
  • 1
  • 17
  • 31
  • 3
    Note that compression support for on-wire data and Feather files is in the project roadmap, but it has not been implemented yet. We also plan to support some kinds of encodings in-memory in addition to dictionary encoding, such as run-length encoding. – Wes McKinney Aug 28 '19 at 15:35