Google Cloud:
I have a pipeline set up where I trigger a workflow when a message is published to a pub sub topic using an eventarc trigger (google.cloud.pubsub.topic.v1.messagePublished).
However lately I've noticed that sometimes the workflow is not triggered randomly even though there were messages posted. It happens once every few days and always happens when multiple messages are published close to each other.
To illustrate I have statistics of 2 messages being published but only one workflow execution around the same time frame.
2 messages published between 9:10PM and 9:11PM
I also see a difference in ack message count compared to the other metrics on the pubsub backing of that eventarc trigger around that time when usually they are all the same value. (don't know if it's related)
has anybody seen this behaviour before? Is it a problem on GCP end? Any help would be appreciated.
As a workaround for now I've set up an alert which goes off if a message is unacked longer than a minute that notifies another pubsub topic which in turn is a second trigger for that workflow and works as a retry whenever it fails.
