Can we read parquet file in Hazelcast jet?

Question

I trying to read parquet file via Hazelcast for that I have written below code which is working fine, but do Hazelcast provide any in-build source to read parquet file?

BatchSource<Object> csvData = SourceBuilder
        .batch("parquet-source", x -> {                               
            try {
    
                ParquetReader<GenericData.Record> reader = AvroParquetReader.<GenericData.Record>builder(new Path("D:/test/1651070287920.parquet")).build();
                return reader;
                
            } catch (Exception e) {
                
                return null;
            }
        })
        .<Object>fillBufferFn((reader, buf) -> {                          
            try {
                GenericRecord record;
            if ((record = reader.read()) != null) {
            
                Map<String, String> map = new HashMap<>();
                for (int i = 0; i < headers[0].length; i++) {
                        String value = record.get(i) == null ? "" : record.get(i).toString();
                        map.put(headers[0][i], value);
                }
            
                
                    if (map != null) {
                        rowcount = rowcount + 1; 
                        buf.add(map);
                    }
            } else {
                buf.close();
                return;
            }
            } catch (Exception e) {
                buf.close();
                return;
            }

        })
        .build();

Please let me know if there is already any source in Hazelcast Jet.

score 2 · Answer 1 · answered May 05 '22 at 09:29

2

Parquet files using Avro for serialization can be read using the Unified File Connector. See also the code sample.

answered May 05 '22 at 09:29

Vlado Schreiner

478
2
5

Thanks for the response, but how can we can get output format in List – user3458271 May 05 '22 at 09:36

František Hartman · Answer 2 · 2022-05-05T10:48:47.367

1

Parquet is supported using the unified file connector

BatchSource<SpecificUser> source = FileSources.files("/data")
  .glob("users.parquet")
  .format(FileFormat.<SpecificUser>parquet())
  .useHadoopForLocalFiles(true)
  .build();

If you don't have a class corresponding to your schema, or your schema is more dynamic and you want to return org.apache.avro.generic.GenericRecord from the source, which you can then map to Map<String, String> you can use the following:

BatchSource<GenericRecord> source = FileSources.files(currentDir + "/target/parquet")
  .glob("file.parquet")
  .option("avro.serialization.data.model", GenericData.class.getName())                                                    
  .useHadoopForLocalFiles(true)
  .format(FileFormat.parquet())
  .build();

edited May 05 '22 at 10:48

answered May 05 '22 at 10:18

František Hartman

14,436
2
40
60

Sorry but Do mean after "pipeline.readFrom(source)" method I have to use map and convert it to Map? but cannot we directly use here only while making BatchSource only? – user3458271 May 05 '22 at 10:47
Yes, I mean `pipeline.readFrom(source).map(..)`. There is really no difference, With different steps you can even change parallelism for reading an mapping independently. – František Hartman May 05 '22 at 10:52

Can we read parquet file in Hazelcast jet?

2 Answers2