Consume PubSub in Google cloud function

Question

Base on official document

I try to create a cloud function with "PubSub Pull Subscription" trigger

import base64

def hello_pubsub(event, context):
    """Triggered from a message on a Cloud Pub/Sub topic.
    Args:
         event (dict): Event payload.
         context (google.cloud.functions.Context): Metadata for the event.
    """
    print("This Function was triggered by messageId {} published at {}".format(context.event_id, context.timestamp))

    if 'data' in event:
        name = base64.b64decode(event['data']).decode('utf-8')
    print('"{}" received!'.format(name))
    
    if 'attributes' in event:
        print(event['attributes'])

    if '@type' in event:
        print(event['@type'])

enter image description here

Then I find an article says that "cloud function will send ACK on its invocation", which is consistent with the official document.

However, when the cloud function finishes processing the PubSub message, "Unacked message count" do not decrease (as shown in the image above)

Hence, I try google-cloud-pubsub on local

subscription_path = subscriber.subscription_path(PROJECT, SUBSCRIPTION)
response = subscriber.pull(subscription_path, max_messages=5)

for msg in response.received_messages:
    print("Received message:", msg.message.data)

ack_ids = [msg.ack_id for msg in response.received_messages]
subscriber.acknowledge(subscription_path, ack_ids)

In this way, the number of message count decrease successfully. enter image description here

My questions are:

Am I missing something in my cloud function script?
How can I actually "consume" the PubSub message in my cloud function?

Any suggestion is appreciated, thank you.

guillaume blaquiere · Accepted Answer · 2020-07-23T07:58:15.253

9

With PubSub, you have publisher, that publish messages into a topic. The message is duplicated in each existing subscription (created on a topic). At the end, subscribers can listen subscription.

So, here, you have have 1 topic, and 1 pull subscription. You also have a Cloud Functions that you deploy on a topic (in gcloud cli, param --trigger-topic=myTopic). ON A TOPIC, not on a subscription.

Go back to the subscription page, you should see 2 subscriptions: Your pull subscription, and a push subscription to a strange endpoint

Therefore, your message is published in the 2 subscriptions. If you look your Pull subscription, nothing consume the message in it, except your code on local. The logs in the cloud functions should show the correct message processing.

Is it clearer?

EDIT

To be precise on your case:

Your Cloud Functions can't ack the messages in the pull subscription, because it is not connected to it
Your Cloud Functions process and ack messages published in its own subscription.

edited Jul 23 '20 at 07:58

answered Jul 22 '20 at 09:54

guillaume blaquiere

66,369
2
47
76

Thank you for the clarification about 'topic' & 'subscription' If I understand you correctly, you mean the way I process PubSub message in cloud function might be wrong, so that I cannot make "Unacked message count" decrease? – han shih Jul 23 '20 at 05:52
I updated my answer. Look at your subscriptions, you should have 2 – guillaume blaquiere Jul 23 '20 at 07:58
1

Oh, I finally got the point! Cloud function ACK messages from Push Subscription (the "strange endpoint" as you mentioned) In the meanwhile, the same messages are sent to Pull Subscription. So I saw the increasing "Unacked message count" of Pull Subscription. One way to reduce the count is to actively pull the messages from PubSub. (the sample code I tried on local) Please correct me if I am wrong. – han shih Jul 23 '20 at 08:34
Yes correct. But because you already process the message from the "push" subscription, do you really need your "pull" subscription? – guillaume blaquiere Jul 23 '20 at 09:35
You are right. Maybe I should remove that subscription. – han shih Jul 24 '20 at 00:28

Consume PubSub in Google cloud function

1 Answers1

Linked