Data enrichment for multiple pubsub topics

Question

I have 2 topics:

player_info_topic example message:

{"id": 1, "name": "Sandy"}

seating_arrangement_topic example message:

{"id": 1, "seat": 2}

Is there a way to match these messages in gcp, cloud dataflow maybe then publish to another pubsub topic?

score 2 · Answer 1 · answered Sep 28 '22 at 10:19

In streaming, joins can only be done if you apply a window to the PCollections to be joined. The window has to be either a FixedWindow or a SlidingWindow.

You could read from the two topics, and then add a key to each element with the id in the two resulting PCollections. Then add the window, and apply the CoGroupByKey. Then generate the output in the format that you want to write to Pubsub, and send it to Pubsub.

Have a look at the examples for Python, for Java and the section about CoGroupByKey in the Beam Programming Guide

Data enrichment for multiple pubsub topics

1 Answers1