1

Currently, I create a service in Cloud Run to retrain ML models. The service will do the retrain process when there is an event from BigQuery called google.cloud.bigquery.v2.JobService.InsertJob. I use the EventArc in GCP to trigger the Retrain Service when that event happened. But, there is a problem. The trigger request to the service multiple times in one event. So, sometimes when the retraining process is done, the trigger requests the service again, and then the retraining process is active again. Is there something that I missed? Picture bellow is my EventArc setup.

enter image description here

As we can see in this picture that there are other requests while the first request is in process. enter image description here

ferdianm10
  • 55
  • 4

1 Answers1

1

Eventarc is backed on PubSub. By default, and if you don't hack the default Eventarc configuration, the delivery timeout is set to 10s. (you can update manually the pubsub subscription created by eventarc. The Eventarc engineering team is aware of that not customizable parameter)

That's why, you should have a retry every 10s.

You have 2 solutions to that:

  • Either create an async process. I mean receive the PubSub message (tbe eventarc event), ack it immediately, and, in background, run your retrain
  • Or (not my preferred way), update the eventarc pubsub subscription and set the message retention duration to 5 seconds.
guillaume blaquiere
  • 66,369
  • 2
  • 47
  • 76