2

I'm using azure SDK, avro-parquet and hadoop libraries to read a parquet file from Blob Container. Currently, I'm downloading file to the temp file, and then create a ParquetReader.

try (InputStream input = blob.openInputStream()) {
                Path tmp = Files.createTempFile("tempFile", ".parquet");

                Files.copy(input, tmp, StandardCopyOption.REPLACE_EXISTING);
                IOUtils.closeQuietly(input);
                InputFile file = HadoopInputFile.fromPath(new org.apache.hadoop.fs.Path(tmp.toFile().getPath()),
                        new Configuration());
                ParquetReader<GenericRecord> reader = AvroParquetReader.<GenericRecord> builder(file).build();

                GenericRecord record;
                while ((record = reader.read()) != null) {
                    recordList.add(record);
                }
            } catch (IOException | StorageException e) {
                log.error(e.getMessage(), e);
            }

I want to read this file using inputStream from azure blob item, without downloading it to my machine. There's such way for S3 ( Read parquet data from AWS s3 bucket), but does this possibility exist for Azure?

Limmy
  • 697
  • 1
  • 13
  • 24

1 Answers1

3

Find out how to do that. 

 StorageCredentials credentials = new StorageCredentialsAccountAndKey(accountName, accountKey);
 CloudStorageAccount connection = new CloudStorageAccount(credentials, true);
 CloudBlobClient blobClient = connection.createCloudBlobClient();
 CloudBlobContainer container = blobClient.getContainerReference(containerName);

 CloudBlob blob = container.getBlockBlobReference(fileName);

 Configuration config = new Configuration();
 config.set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem");
 config.set("fs.azure.sas.<containerName>.<accountName>.blob.core.windows.net", token);
 URI uri = new URI("wasbs://<containerName>@<accountName>.blob.core.windows.net/" + blob.getName());
 InputFile file = HadoopInputFile.fromPath(new org.apache.hadoop.fs.Path(uri),
                config);
 ParquetReader<GenericRecord> reader = AvroParquetReader.<GenericRecord> builder(file).build();

 GenericRecord record;
 while ((record = reader.read()) != null) {
     System.out.println(record);
 }
 reader.close();
Limmy
  • 697
  • 1
  • 13
  • 24
  • Can you add the imports here? I am confused from where the "AvroParquetReader" is coming? – QualityMatters Mar 15 '22 at 09:37
  • org.apache.parquet.avro.AvroParquetReader from parquet-avro lib – Limmy Mar 22 '22 at 16:18
  • Can you share the full working code with pom dependencies and all imports, as I am struggling with it and not getting solution anywhere. It would help a lot. Thanks – QualityMatters Mar 23 '22 at 04:47
  • I have shared my code in this question. https://stackoverflow.com/questions/71592646/azure-java-file-not-found-exception-when-trying-to-access-file-from-azure – QualityMatters Mar 23 '22 at 20:28