PubSub message streaming interrupted - 502:Bad Gateway

Question

I have a backend that consistently pulls message from a PubSub subscription and this has been working well up until a couple of weeks ago. For the past 2-3 weeks the streaming process frequently stopped and refused to restart. When I looked into the logs I found that the streaming process runs into a 502:Bad Gateway error :

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "502:Bad Gateway"
    debug_error_string = "{"created":"@1659148551.896603077","description":"Error received from peer ipv4:64.233.184.95:443","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"502:Bad Gateway","grpc_status":14}"

which then lead to a RetryError :

google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 502:Bad Gateway

When that happens I try to reinitialize the streaming process but it keeps failing over and over and the only way for it to restart is for me to kill the python process and execute it again.

Here is the code that tries to restart the process automatically :

def start_pubsub_streaming(pubsub_pulling_thread):
    """
    Start the streaming thread. If an exception arises then stop the
    streaming it
    """
    try:
        # When `timeout` is not set, result() will block indefinitely,
        # unless an exception is encountered first.
        _LOGGER.info(f"Starting streaming process")
        pubsub_pulling_thread.result()
        _LOGGER.info(f"Listening for messages on {SUBSCRIPTION_PATH}..\n")
    except Exception as e:
        _LOGGER.warning('Exception raised while pulling data, see below', exc_info=True)
        stop_pubsub_streaming(pubsub_pulling_thread)
    finally :
        _LOGGER.info(f"Sleeping for 5 minutes before restarting the streaming")
        time.sleep(60*5)
        _LOGGER.info(f"Proceeding to streaming restart")
        start_pubsub_streaming(pubsub_pulling_thread)


def stop_pubsub_streaming(pubsub_pulling_thread):
    """ Stops the streaming thread """
    _LOGGER.info('Stopping the streaming process ...')
    pubsub_pulling_thread.cancel()  # Trigger the shutdown.
    pubsub_pulling_thread.result()  # Block until the shutdown is complete.
    _LOGGER.info('Streaming process stoped.')

My two main concerns are :

Why do I keep having these errors, and how to I stop it from happening ?
How do I programmatically restart the streaming process if I were to run into these again.

It is worth noting that I am using Python 3.7.3 and google-cloud-pubsub==2.9.0.

Also here is how I am running the streaming process in a separate thread (simplified for the post).

def create_pubsub_streaming_object():
    """ Create an asynchronous PubSub streaming thread"""
    # Limit the subscriber to only have ten outstanding messages at a time.
    flow_control = pubsub_v1.types.FlowControl(max_messages=10)
    streaming_pull_future = _SUBSCRIBER.subscribe(
        SUBSCRIPTION_PATH,
        callback=store_messages,
        flow_control=flow_control
    )
    return streaming_pull_future

_PUBSUB_STREAMING_OBJECT = create_pubsub_streaming_object()
# Creating the streaming thread
pull_thread = threading.Thread(
        target=start_pubsub_streaming,
        args=[_PUBSUB_STREAMING_OBJECT]
    )
pull_thread.daemon = True
pull_thread.name = 'Data pulling thread'
pull_thread.start()

PubSub message streaming interrupted - 502:Bad Gateway

0 Answers0