Data Compression using Trino Iceberg connector

Asked Nov 08 '22 at 07:03

Active Nov 08 '22 at 07:31

Viewed 147 times

I have a situation that I am investigating, I am looking to compress parquet files that are sitting on AWS S3, the typical sizes for those files are 90-100 MB, the reason for that is to reduce I/O operations. I am looking to use iceberg connector to compress those files.

The problem I have is that this is experimental and I don't have a best way of going about this, are the any approaches I could adopt on using iceberg connectors?

asked Nov 08 '22 at 07:03

user3476582

I'm not familiar with Trino,but [spark produce][1] is a good option if you can use spark, which can help you rewrite files with a different strategy. Examples: ``` CALL catalog_name.system.rewrite_data_files('db.sample') ``` [1]: https://iceberg.apache.org/docs/latest/spark-procedures/ – liliwei Nov 08 '22 at 07:33
Using Iceberg just for compression is a great overkill – shay__ Nov 08 '22 at 08:42
why? what are the other alternatives? – user3476582 Nov 08 '22 at 18:09
Are you referring to **compression** or **compaction**? – shay__ Nov 13 '22 at 09:07
I'm referring to compression – user3476582 Nov 14 '22 at 07:16

Data Compression using Trino Iceberg connector

0 Answers0