0

Problem: I'm building a mobile game app with real-time scoring. Each time a player performs an action, it sends a message to Pub/Sub with the following keys: {"event_ts","event_id", "player_id", "score"}

As soon as the Pub/Sub message is received, I want to use Beam to do a transformation to enrich the message based on that "player_id"'s average "score" in the last 5 minutes.

In other words, the output should be: {"event_ts", "event_id", "player_id", "score", "avg_score_last5min"}

For example, there are two players ("A, B"):

"event_ts":2020-01-01 12:01:00, "event_id":11, "score":10, "player_id":A-------> "event_ts":2020-01-01 12:01:00, "event_id":11, "avg_score_last5min":10, "player_id":A
"event_ts":2020-01-01 12:01:00, "event_id":12, "score":20, "player_id":B-------> "event_ts":2020-01-01 12:01:00, "event_id":12, "avg_score_last5min":20, "player_id":B
"event_ts":2020-01-01 12:02:00, "event_id":13, "score":30, "player_id":A-------> "event_ts":2020-01-01 12:02:00, "event_id":13, "avg_score_last5min":20, "player_id":A
"event_ts":2020-01-01 12:03:00, "event_id":14, "score":20, "player_id":A-------> "event_ts":2020-01-01 12:03:00, "event_id":14, "avg_score_last5min":20, "player_id":B
"event_ts":2020-01-01 12:03:00, "event_id":15, "score":10, "player_id":B-------> "event_ts":2020-01-01 12:03:00, "event_id":15, "avg_score_last5min":15, "player_id":A
"event_ts":2020-01-01 12:04:00, "event_id":16, "score":20, "player_id":A-------> "event_ts":2020-01-01 12:04:00, "event_id":16, "avg_score_last5min":20, "player_id":A
"event_ts":2020-01-01 12:05:00, "event_id":17, "score":10, "player_id":A-------> "event_ts":2020-01-01 12:05:00, "event_id":17, "avg_score_last5min":18, "player_id":A
"event_ts":2020-01-01 12:09:00, "event_id":18, "score":10, "player_id":A-------> "event_ts":2020-01-01 12:09:00, "event_id":18, "avg_score_last5min":15, "player_id":A

As a new Pub/Sub Message is received, I want to do a transformation to calculate "the average score in the last 5 min window" for that customer at that event time.

How often I want the outputs? I need one output per event. That is, the output need to emit as soon as event received (close to real-time).

I have already read all of these [samples][1] but none of those are work for my case. [1]: https://beam.apache.org/get-started/mobile-gaming-example/

Sid
  • 1
  • 1

1 Answers1

0

What you are trying to do is grouping elements into fixed-length (5-minute) session windows but assigning the event time of the next incoming element to be the end of each window.

I don't think it's feasible in Beam. The closest thing is Custom Windows that allows you to create windows by using an element's event time + gap in the future.

So instead of avg_score_last5min, you can do avg_score_next5min.

Additionally, if you just want a simple moving average every 5 minute, you can use sliding windows, see Calculating moving average over the last 24h with Apache Beam

ningk
  • 1,298
  • 1
  • 7
  • 7