Does iceberg support writing data into GCS? Because for the iceberg's atomicity to work according to https://iceberg.apache.org/java-api-quickstart/, GCS should support atomic rename, however from https://cloud.google.com/storage/docs/gsutil/commands/mv GCS renames are not atomic. What are the workarounds to handle atomicity if iceberg doesn't support GCS
Asked
Active
Viewed 638 times
1 Answers
0
The Hadoop table implementation is the only place where rename is used, and it requires a file system that supports atomic rename. If you're using an object store like S3 or GCS, then you should be using the HMS implementation or a custom catalog instead of Hadoop tables.
"Tables do not require rename, except for tables that use atomic rename to implement the commit operation for new metadata files."
Everything should work with GCS without modification as far as I know.
There is no native atomic rename in GCS, it requires a move + delete. You can refer to the page https://iceberg.apache.org/spec/#mvcc-and-optimistic-concurrency.
Anurag Sharma
- 2,409
- 2
- 16
- 34