Stack Overflow
Stack Overflow

Questions tagged [iceberg]

Apache Iceberg is an open table format for huge analytic datasets. Iceberg adds tables to Presto and Spark that use a high-performance format that works just like a SQL table. Use this tags for any questions relating to support for or usage of Iceberg.

134 questions
0
votes
1 answer

Trino iceberg connector "getTablesWithParameter for GlueHiveMetastore is not implemented"

I'm running trino on EMR version 6.5 and I have added the iceberg connector for the trino and I want it to use a glue catalog. These are the configuration under the iceberg.properties connector.name=iceberg iceberg.file-format=PARQUET hive.metastore…
taraf
  • 777
  • 2
  • 10
  • 28
0
votes
1 answer

How to run Apache Flink with Hive metastore locally to test Apache Iceberg

I would like to fiddle a bit around with Apache Flink and Apache Iceberg and test this on a local machine. I read through the documentation, but I'm still not sure what has to be setup locally to make this run. What I already did is that I have a…
Lothium
  • 49
  • 7
0
votes
1 answer

Iceberg: How to quickly traverse a very large table

I'm new to iceberg, and i have a question about query big table. We have a Hive table with a total of 3.6 million records and 120 fields per record. and we want to transfer all the records in this table to other databases, such as pg, kafak,…
xujin
  • 1
  • 1
0
votes
0 answers

Issue with Apache Hudi Update and Delete Operation on Parquet S3 File

Here I am trying to simulate updates and deletes over a Hudi dataset and wish to see the state reflected in Athena table. We use EMR, S3 and Athena services of AWS. Attempting Record Update with a withdrawal object withdrawalID_mutate =…
jishmisc28
  • 9
  • 5
0
votes
1 answer

Apache Spark UDF: Accessing Iceberg

I am trying to access an Iceberg table from within a Spark Java UDF, but I am getting an error when running the first SQL statement in the UDF. Here is how I create the Spark session in the UDF: SparkSession spark = …
Martin Dubuc
  • 96
  • 6
0
votes
2 answers

How to unstruct a Struct in SQL

I have a struct in a table (Iceberg Database format) and I would like to expand all of the children of the struct. The normal query would look like : SELECT base.el1, base.el2, base.el3 FROM myTable Instead of that, I would like to have a…
Pitchkrak
  • 340
  • 1
  • 3
  • 11
0
votes
1 answer

Write Flink DataStream to Iceberg Table：NoSuchMethodError: org.apache.parquet.schema.Types$PrimitiveBuilder.as

I try to write a flink datastream to a iceberg table, as below: ''' val kafkaStream = new KafkaDataSource(parameter, new PacketSchema).getStream(env) val dataStream = kafkaStream.flatMap(new…
K. Chen
  • 36
  • 3
0
votes
1 answer

Iceberg GCS and Consistency

Does iceberg support writing data into GCS? Because for the iceberg's atomicity to work according to https://iceberg.apache.org/java-api-quickstart/, GCS should support atomic rename, however from…
coderatcloud9
  • 85
  • 1
  • 1
  • 7
0
votes
1 answer

Iceberg is not working when writing AVRO from spark

We are encountering the following error when appending AVRO files from GCS to table. The avro files are valid but we use deflated avro, is that a concern? Exception in thread "streaming-job-executor-0" java.lang.NoClassDefFoundError:…
coderatcloud9
  • 85
  • 1
  • 1
  • 7
0
votes
1 answer

Iceberg's FlinkSink doesn't update metadata file in streaming writes

I have been trying to use Iceberg's FlinkSink to consume the data and write to sink. I was successful in fetching the data from kinesis and I see that the data is being written into the appropriate partition. However, I don't see the metadata.json…
Sai Krishna
  • 3
  • 2
0
votes
1 answer

SparkSQL DELETE command doesn't delete one single row in Apache Iceberg, does it?

I use Spark SQL 3.0 with scala_2.12. I insert data into the iceberg table and read data from the tabel successfully.when i tried to delete one wrong record from the tabel by spark SQL , the log shows exception . The issue 1444 of apache iceberg in…
harryboot
  • 13
  • 1
  • 6
0
votes
1 answer

Can't write data into the table by Apache Iceberg

i'm trying to write simple data into the table by Apache Iceberg 0.9.1, but error messages show. I want to CRUD data by Hadoop directly. i create a hadooptable , and try to read from the table. after that i try to write data into the table . i…
harryboot
  • 13
  • 1
  • 6
0
votes
1 answer

Standalone hive metastore with Iceberg and S3

I'd like to use Presto to query Iceberg tables stored in S3 as parquet files, therefore I need to use Hive metastore. I'm running a standalone hive metastore service backed by MySql. I've configured Iceberg to use Hive catalog: import…
dmgcodevil
  • 629
  • 1
  • 7
  • 23
-2
votes
2 answers

Iceberg as external table in Snowflake

When is GA planned for Iceberg as an external table in Snowflake? Last I checked it was in private preview, I was hoping it to be available by now.
Ankur Lodha
  • 11
  • 2
1 2 3
…
8
9