I have an Apache Beam streaming project that uses Combine.perKey(), I need to be able to merge entities from my admin tool (to point one entity to another one), how to combine two keys with calculated data in Beam? It's easy to do it for the new messages, but how to combine already calculated data?
Asked
Active
Viewed 195 times
0
Oleksandr Vetoshkin
- 309
- 2
- 10
-
What about storing the calculated data into PCollections and then combine them again? – Jose Gutierrez Paliza May 13 '22 at 18:55
-
@JoseGutierrezPaliza, yes, but how to do it in the code? – Oleksandr Vetoshkin May 14 '22 at 07:11
1 Answers
0
You can re-key the elements before combine so that the calculated data with two keys you want to merge will have same key.
Yichi Zhang
- 351
- 1
- 5
-
I need to somehow re-key old data too, not just new records – Oleksandr Vetoshkin May 27 '22 at 09:54