2

I created a small program in Python for reading messages from a Pub/Sub subscription. I am using Python 3.7 and google-cloud-pubsub 1.1.0.

My code is very simple:

from google.cloud import pubsub_v1
from google.auth import jwt
import json

service_account_info = json.load(open("service-account-info.json"))
audience_sub = "https://pubsub.googleapis.com/google.pubsub.v1.Subscriber"

credentials_sub = jwt.Credentials.from_service_account_info(
    service_account_info, audience=audience_sub
)

subscriber_ring = pubsub_v1.SubscriberClient(credentials=credentials_sub)

def callback1(message):
    print("In callback!!")
    print(message.data)
    message.ack()

sub_path = "projects/my-project/subscriptions/my-sub"
future = subscriber_ring.subscribe(sub_path, callback=callback1)
future.result()

When the code reaches "future.result()", it hangs there forever and times out 10 minutes later with the error

pubsub 503 failed to connect to all addresses

I already verified that:

  • Pub/Sub is up and running
  • My service account has all the needed permissions. I even tried with my personal Google Cloud account (I am the project owner) with the same results.
  • There are unacked messages in the Topic
  • My network connection is OK

but I cannot make it work. Any ideas?

EDIT: I got some more info from the exception:

om_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/cloud/pubsub_v1/publisher/_batch/thread.py", line 219, in _commit
    response = self._client.api.publish(self._topic, self._messages)
  File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", line 498, in publish
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/api_core/retry.py", line 206, in retry_target
    last_exc,
  File "<string>", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x7fa030891a70>, 


, metadata=[('x-goog-request-params', 'topic=projects/my-project/subscriptions/my-sub'), ('x-goog-api-client', 'gl-python/3.7.6 grpc/1.26.0 gax/1.16.0 gapic/1.2.0')]), last exception: 503 failed to connect to all addresses
OCDev
  • 655
  • 1
  • 7
  • 21
  • Could you attach the full error log? – Stefan Neacsu Feb 05 '20 at 09:55
  • The only error I get is: pubsub 503 failed to connect to all addresses Seems something related to my python environment, if I run the same command through gcloud pubsub subscriptions pull "projects/my-project/subscriptions/my-sub", it works file. I tried creating a new environment with anaconda, but get the same error – OCDev Feb 06 '20 at 00:04
  • Actually I am trying both separate, both fail with the same error. – OCDev Feb 07 '20 at 00:02
  • 503 errors should be retried by the client library, is this failing consistently? Were you able to publish or receive messages at all? – Qiqi Wu Feb 07 '20 at 16:12
  • yes, it keeps failing forever with the same 503 and don't get anything read or published. very weird. – OCDev Feb 08 '20 at 01:59
  • If you checked that Pub/Sub is up and running, I do not believe that the issue is on the server side. I would double check the following [quickstart](https://cloud.google.com/pubsub/docs/quickstart-client-libraries#receive_messages) and pay special attention to the way you are authenticating. Keep me posted. – sllopis Feb 10 '20 at 16:36

1 Answers1

0

It is likely that there is a firewall rule in place or some network configuration that is disallowing/dropping connections to *.googleapis.com (or specifically pubsub.googleapis.com). You can see an example of this with another Google product.

Kamal Aboul-Hosn
  • 15,111
  • 1
  • 34
  • 46