How to manually commit kafka offset after FileIO in apache beam?

Question

I have a FileIO writing a Pcollection<GenericRecord> to files and returns WriteFilesResult<DestinationT>.

I would like to create a DoFn after writing files to commit the offset of written records to kafka but since my offsets are stored in my GenericRecords I can no longer access them in the output of FileIO.

What is the best way to solve this ?

score 1 · Accepted Answer · answered Jun 06 '22 at 16:06

1

For anyone interested, here is how I did:

manually groupbykey records by DestinationT
for each group I get the list of offsets and I create a new key EnrichedDestinationT + flatten the iterable
so the Pcollection before entering FileIO is PCollection<KV<EnrichedDestinationT, GenericRecord>>
in FileIO, the .by() becomes .by(KV::getKey) and the .via() becomes .via(Contextful.fn(KV::getValue), Contextful.fn(this::getSink))

answered Jun 06 '22 at 16:06

Jean Wisser

55
9

very interesting! thanks for sharing : ) this is a good use case – Pablo Jun 14 '22 at 22:31

How to manually commit kafka offset after FileIO in apache beam?

1 Answers1