0

Is there a way to know that Dafaflow job reading from a Pub/Sub topic is hitting a quota Cloud Pub/Sub API Subscriber throughput, kB per minute.

There were no indications of problems in the log, however a pipeline was not able to keep up with backlog of messages from Pub/Sub topic.

Update: I meant to find out programmatically (e.g. via en exception/an error log).

Donald Duck
  • 109
  • 2

1 Answers1

0

As per the Pub/Sub quota limits, there are Subscriber throughput quotas visible on the APIs and services quotas dashboard(Cloud Console > API & Services > Dashboard > Cloud Pub/Sub API > Quota).

If you find that you have hit a limit you may be able to request an increase via the edit option (pencil icon) next to "limits".

Julie
  • 1
  • 1
  • If the aforementioned quota is being hit, please check if there is any explicit error messages in [Stackdriver logging](https://cloud.google.com/logging/docs/view/overview) and if there is not then monitoring the usage using [Stackdriver monitoring API](https://cloud.google.com/monitoring/api/v3/) with [Pub/Sub metrics subscription request count metric](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub) should allow you to monitor the usage of Subscriber throughput. – Julie Nov 07 '19 at 15:05
  • This [overview of quota](https://cloud.google.com/docs/quota) doesn't seem to detail any options of programmatically viewing pub/sub quotas directly. – Julie Nov 07 '19 at 15:07