-1

I am making a dataflow (java api) which will write protobuf object to gcs. Can anyone please help me with how to do this. A sample of code or something would be helpful.

Mikhail Berlyant
  • 165,386
  • 8
  • 154
  • 230
Raman
  • 216
  • 3
  • 15

1 Answers1

1

https://issues.apache.org/jira/browse/BEAM-221 has some thoughts on this.

Wrap AvroIO in a Composite PTransform:

  • Avro schema is just a single byte[].
  • On read, convert from byte[] to the user's proto.
  • On write, convert from proto to byte[].

In this way, ProtoIO would just be a simple wrapper for AvroIO.Read+ParDo (read) or ParDo+AvroIO.Write.

Sam McVeety
  • 3,194
  • 1
  • 15
  • 38