Iceberg table does not see the generated Parquet file

Question

In my use case, the table in Iceberg format is created. It only receives APPEND operations as it is about recording events in a time series stream. To evaluate the use of the Iceberg format in this use-case, I created a simple Java program that creates a set of 27600 lines. Both the metadata and the parquet file were created but I can't access them via the Java API (https://iceberg.apache.org/docs/latest/java-api-quickstart/). I'm using HadoopCatalog and FileAppender<GenericRecord>. It is important to say that I can read the Parquet file created using pyarrow and datafusion modules via Python 3 script, and it is correct!

I believe that the execution of some method in my program that links the generated Parquet file to the table created in the catalog must be missing.

NOTE: I'm only using Apache Iceberg's Java API in version 1.0.0

There is an org.apache.iceberg.Transaction object in the API that accepts an org.apache.iceberg.DataFile but I haven't seen examples of how to use it and I don't know if it's useful to solve this problem either.

See the program below:

import org.apache.hadoop.conf.Configuration;
import org.apache.iceberg.*;
import org.apache.iceberg.catalog.Catalog;
import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.data.GenericRecord;
import org.apache.iceberg.data.parquet.GenericParquetWriter;
import org.apache.iceberg.hadoop.HadoopCatalog;
import org.apache.iceberg.io.FileAppender;
import org.apache.iceberg.parquet.Parquet;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.types.Types;

import java.io.File;
import java.io.IOException;
import java.time.LocalDate;
import java.time.temporal.ChronoUnit;
import java.util.List;

import static org.apache.iceberg.types.Types.NestedField.optional;
import static org.apache.iceberg.types.Types.NestedField.required;

public class IcebergTableAppend {
    public static void main(String[] args) {
        System.out.println("Appending records ");
        Configuration conf = new Configuration();
        String lakehouse = "/tmp/iceberg-test";
        conf.set(CatalogProperties.WAREHOUSE_LOCATION, lakehouse);
        Schema schema = new Schema(
                required(1, "hotel_id", Types.LongType.get()),
                optional(2, "hotel_name", Types.StringType.get()),
                required(3, "customer_id", Types.LongType.get()),
                required(4, "arrival_date", Types.DateType.get()),
                required(5, "departure_date", Types.DateType.get()),
                required(6, "value", Types.DoubleType.get())
        );
        PartitionSpec spec = PartitionSpec.builderFor(schema)
                .month("arrival_date")
                .build();
        TableIdentifier id = TableIdentifier.parse("bookings.rome_hotels");
        String warehousePath = "file://" + lakehouse;
        Catalog catalog = new HadoopCatalog(conf, warehousePath);
        // rm -rf  /tmp/iceberg-test/bookings
        Table table = catalog.createTable(id, schema, spec);
        List<GenericRecord> records = Lists.newArrayList();
        // generating a bunch of records
        for (int j = 1; j <= 12; j++) {
            int NUM_ROWS_PER_MONTH = 2300;
            for (int i = 0; i < NUM_ROWS_PER_MONTH; i++) {
                GenericRecord rec = GenericRecord.create(schema);
                rec.setField("hotel_id", (long) (i * 2) + 10000);
                rec.setField("hotel_name", "hotel_name-" + i + 1000);
                rec.setField("customer_id", (long) (i * 2) + 20000);
                rec.setField("arrival_date",
                        LocalDate.of(2022, j, (i % 23) + 1)
                                .plus(1, ChronoUnit.DAYS));
                rec.setField("departure_date",
                        LocalDate.of(2022, j, (i % 23) + 5));
                rec.setField("value", (double) i * 4.13);
                records.add(rec);
            }
        }
        File parquetFile = new File(
                lakehouse + "/bookings/rome_hotels/arq_001.parquet");
        FileAppender<GenericRecord> appender = null;
        try {
            appender = Parquet.write(Files.localOutput(parquetFile))
                    .schema(table.schema())
                    .createWriterFunc(GenericParquetWriter::buildWriter)
                    .build();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        try {
            appender.addAll(records);
        } finally {
            try {
                appender.close();
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        }
    }
}

score 2 · Answer 1 · answered Oct 22 '22 at 21:01

I found out how to fix the Java program.

Just add the lines below to the end of the main method

PartitionKey partitionKey = new PartitionKey(table.spec(), table.schema());
DataFile dataFile = DataFiles.builder(table.spec())
        .withPartition(partitionKey)
        .withInputFile(localInput(parquetFile))
        .withMetrics(appender.metrics())
        .withFormat(FileFormat.PARQUET)
        .build();
 Transaction t = table.newTransaction();
 t.newAppend().appendFile(dataFile).commit();
 // commit all changes to the table
 t.commitTransaction();

Also add to your POM file the dependency below

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-core</artifactId>
    <version>3.3.4</version>
</dependency>

This avoids the runtime error shown below:

java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.lib.input.FileInputFormat

Iceberg table does not see the generated Parquet file

1 Answers1