How to get/add GCS File user-defined metadata using Apache Beam library [org.apache.beam.sdk.io.*]

Question

I'm setting up a Dataflow pipeline, in which one of the action is to get/add the metadata[User-provided metadata] of a GCS file.

In a standalone java app I used below method to get the metadata which is from StorageObject class but not finding something similar method/api in Apache Beam library. Any pointers will be really helpful.

//Below code is from StorageObject.java
com.google.api.services.storage.model.StorageObject
//.....
public java.util.Map<String, java.lang.String> getMetadata() {
     return metadata;
}

Are you able to make use of the client library directly within a DoFn as user code? — Reza Rokni, Jun 27 '19 at 05:47
I used com.google.cloud.storage.Storage to get the metadata of a GCS file. — Bala, Jul 02 '19 at 05:54

score 1 · Accepted Answer · answered Jul 02 '19 at 05:50

I used following code to get the metadata from GCS and it works good in DataFlow pipeline.

import com.google.cloud.storage.{BlobId, BlobInfo, Storage, StorageOptions}

val storage: Storage = StorageOptions.getDefaultInstance.getService
val blobId: BlobId = BlobId.of("bucket", "filename")
val srcMap: Map[String, String] = storage.get(blobId).getMetadata

How to get/add GCS File user-defined metadata using Apache Beam library [org.apache.beam.sdk.io.*]

1 Answers1