5

Is it possible to trigger an HTTP cloud function in response to a pubsub message?

When editing a subscription, google makes it possible to push the message to an HTTPS endpoint, but for abuse reasons one has to be able to prove that you own the domain in order to do this, and of course you can't prove that you own google's own *.cloudfunctions.net domain which is where they get deployed.

The particular topic I'm trying to subscribe to is a public one, projects/pubsub-public-data/topics/taxirides-realtime. The answer might be use a background function rather than HTTP triggered, but that doesn't work for different reasons:

gcloud functions deploy echo --trigger-resource projects/pubsub-public-data/topics/taxirides-realtime --trigger-event google.pubsub.topic.publish ERROR: gcloud crashed (ArgumentTypeError): Invalid value 'projects/pubsub-public-data/topics/taxirides-realtime': Topic must contain only Latin letters (lower- or upper-case), digits and the characters - + . _ ~ %. It must start with a letter and be from 3 to 255 characters long.

This seems to indicate this is only permitted on topics I own, which is a strange limitation.

FrobberOfBits
  • 17,634
  • 4
  • 52
  • 86
  • I don't understand what you're trying to accomplish here. – Doug Stevenson Jul 29 '18 at 02:37
  • I'm trying to take data from a public pubsub topic (`projects/pubsub-public-data/topics/taxirides-realtime`) and put it into a google cloud function. I can't do this with an HTTP triggered cloud function because I don't own the domain, and I can't do it with a background function because it doesn't permit creating a subscription to a topic I don't control. @DougStevenson – FrobberOfBits Jul 29 '18 at 14:26
  • I don't think there's such a thing as a "public pubsub topic". You can only get messages from a topic from within the project where the topic has been created. – Doug Stevenson Jul 29 '18 at 16:04
  • @DougStevenson there is such a thing as a public pubsub topic -- the topic listed above is one such. https://stackoverflow.com/questions/38861321/public-google-cloud-pub-sub-topics – FrobberOfBits Jul 29 '18 at 17:35
  • Could you edit your question to point to the documentation for public pubsub topics and how they're intended to work? – Doug Stevenson Jul 29 '18 at 17:38
  • Possible duplicate of [Google Pubsub subscriber that does a http PUSH to a cloud function (in same project or another project)](https://stackoverflow.com/questions/49696913/google-pubsub-subscriber-that-does-a-http-push-to-a-cloud-function-in-same-proj) – A.Queue Jul 30 '18 at 16:02
  • This looks like the answer to your question https://stackoverflow.com/questions/49696913. – A.Queue Jul 30 '18 at 16:03

3 Answers3

3

It is possible to publish from a pub/sub topic to a cloud function. I was looking for a way to publish messages from a topic in project A to a function in project B. This was not possible with a regular topic trigger, but it is possible with http-trigger. Overall steps to follow:

  • Creata a http-triggered function in project B.
  • Create a topic in project A.
  • Create a push subscription on that topic in project A.
  • Domain verification

Push subscription

enter image description here

Here we have to fill in three things: the endpoint, the audience and the service account under which the function runs.

  • Push Endpoint: https://REGION-PROJECT_ID.cloudfunctions.net/FUNC_NAME/ (slash at end)
  • Audience: https://REGION-PROJECT_ID.cloudfunctions.net/FUNC_NAME (no slash at end)
  • Service Account: Choose a service account under which you want to send the actual message. Be sure the service account has the "roles/cloudfunctions.invoker" role on the cloud function that you are sending the messages to. Since november 2019, http-triggered functions are automatically secured because AllUsers is not set by default. Do not set this property unless you want your http function to be public!

Domain verification

Now you probably can't save your subscription because of an error, that is because the endpoint is not validated by Google. Therefore you need to whitelist the function URL at: https://console.cloud.google.com/apis/credentials/domainverification?project=PROJECT_NAME.

Following this step will also bring you to the Google Search Console, where you would also need to verify you own the endpoint. Sadly, at the time of writing this process cannot be automated.

  • Next we need to add something in the lines of the following (python example) to your cloud function to allow google to verify the function:

    if request.method == 'GET':
    return '''
        <html>
            <head>
                <meta name="google-site-verification" content="{token}" />
            </head>
            <body>
            </body>
        </html>
    '''.format(token=config.SITE_VERIFICATION_CODE)

Et voila! This should be working now.

Sources: https://github.com/googleapis/nodejs-pubsub/issues/118#issuecomment-379823198, https://cloud.google.com/functions/docs/calling/http

Cloudkollektiv
  • 11,852
  • 3
  • 44
  • 71
  • "Now you probably can't save your subscription because of an error, that is because the endpoint is not validated by Google. Therefore you need to whitelist the function URL at: API & Services > Credentials > Domain Verification." As the question states, I don't see how this is possible because you can't prove ownership of cloudfunctions.net. – Ellis Oct 08 '19 at 10:18
  • You prove ownership by including the google-site-verification in a GET request, which I clearly stated in my answer. – Cloudkollektiv Oct 08 '19 at 18:59
  • As on 2022, it this domain validation still needed? – jcvegan Jul 19 '22 at 02:40
2

Currently, Cloud Functions does not allow one to create a function that receives messages for a topic in a different project. Therefore, specifying the full path including "projects/pubsub-public-data" does not work. The gcloud command to deploy a Cloud Function for a topic expects the topic name only (and not the full resource path). Since the full resource path contains the "/" character, it is not a valid specification and results in the error you see.

Kamal Aboul-Hosn
  • 15,111
  • 1
  • 34
  • 46
-1

The error you are getting seems to be that you are misspelling something in the gcloud command you are issuing.

ERROR: gcloud crashed (ArgumentTypeError): Invalid value 'projects/pubsub-public-data/topics/taxirides-realtime': Topic must contain only Latin letters (lower- or upper-case), digits and the characters - + . _ ~ %. It must start with a letter and be from 3 to 255 characters long

Are you putting a newline character in the middle of the command?

Jason Ganetsky
  • 411
  • 2
  • 7