1

I am trying to utilize pub/sub service and I noticed in my dashboard following error code.

enter image description here

Here link what is code 503

Is there anything that allow me to prevent that?

-Askar

kostix
  • 51,517
  • 14
  • 93
  • 176
Askar
  • 513
  • 5
  • 22

3 Answers3

3

As explained in the documentation link about Error Codes that you shared, the HTTP code 503 ("UNAVAILABLE") is returned when the Pub/Sub service was not able to process a request. In general, one could say that these types of errors tend to be transient, and there's no way to avoid them, you can just work around them following a retry strategy such as the one I will comment shortly.

The Google Cloud Pub/Sub SLA shows the guaranteed uptime for this service. As you can see, it is not 100%, as transient errors may happen, which should not disturb your service greatly, considering that you follow the recommended practice of implementing a retry strategy with exponential backoff.

This documentation page shows an example implementation of an Exponential Backoff retry strategy. This example is for Google Cloud Storage, but it can (and should) be applied to any other similar service. It consists in retrying the failed Pub/Sub requests with an increasing backoff in order to increase the probability of a request being successful. This is a recommended best practice and the recommended approach to overcome transient issues.

dsesto
  • 7,864
  • 2
  • 33
  • 50
  • Thanks so much! I found some discussion on pubsub channel in the slack. From Slack: Doug Hoard [8:56 AM] We use a 5 minute ack deadline timeout. We start processing the message, call the api, etc. with Fibonacci back-off for 10 times for retry. If we don’t succeed at that point, we just don’t NACK the message, so it will get redelivered once the ack deadline timeout expires. – Askar Apr 03 '18 at 19:14
  • I am glad I could help. I understand your question is now solved, and you will use a backoff retry implementation. If that is the case, feel free to [accept my answer](https://stackoverflow.com/help/someone-answers) so that the question is seen as solved by the community. Thanks! – dsesto Apr 04 '18 at 07:44
1

StreamingPull has a 100% error rate.

StreamingPull streams are always terminated with a non-OK status(HTTP 503). Note that, unlike in regular RPCs, the status here is simply an indication that the stream has been broken, not that requests are failing.

https://cloud.google.com/pubsub/docs/pull#streamingpull_has_a_100_error_rate_this_is_to_be_expected

Febin
  • 161
  • 2
  • 3
0

I was getting the same errors when subscribing. It stopped after I set the timeout.

timeout = 600 # it was previously None
# Wrap subscriber in a 'with' block to automatically call close() when done.
    with subscriber:
        try:
            # When `timeout` is not set, result() will block indefinitely,
            # unless an exception is encountered first.
            listener_streaming_pull_future.result(timeout=timeout)
        except TimeoutError:
            listener_streaming_pull_future.cancel()  # Trigger the shutdown.
            listener_streaming_pull_future.result()  # Block until the shutdown is complete.
Titu
  • 176
  • 7