3

I am testing out a very basic Pub/Sub subscription. I have the push endpoint set to an App I have deployed through a Python Flex service in App Engine. The service is in a project with Identity-Aware Proxy enabled. The IAP is configured to allow through users authenticated with our domain.

I do not see any of the push requests being processed by my app.

I turned off the IAP protection and then I see that the requests are processed. I turn it back on and they are no longer processed.

John Hanley
  • 74,467
  • 6
  • 95
  • 159
Jefila
  • 131
  • 1
  • 11
  • Pub/Sub push to an IAP-protected App Engine isn't currently supported. There's a Feature Request created for it here: https://issuetracker.google.com/issues/69892982. Possible workarounds are mentioned in this SO post (it's old but still holds true): https://stackoverflow.com/a/47233259/4926605 – LundinCast Sep 06 '19 at 07:51

1 Answers1

6

Note: This answer uses BETA commands and features.

To enable IAP enabled App Engine access to Pub/Sub push notifications:

  • Enable Pub/Sub to create Identity Tokens
  • Create a service account that Pub/Sub will use for its identity to IAP
  • Create a push Pub/Sub subscription with the service account
  • Add the Pub/Sub service account email address to IAP

Allow the Pub/Sub service (Service Agent) to create Identity Tokens on behalf of a service account:

gcloud projects add-iam-policy-binding PROJECT-ID \
     --member=serviceAccount:service-PROJECT-NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com \
     --role=roles/iam.serviceAccountTokenCreator

Create a service account that Pub/Sub will use for its identity to IAP:

gcloud iam service-accounts create pubsub-invoker \
     --display-name "Pub/Sub Invoker Service Account"

Create a push Pub/Sub subscription with the service account:

gcloud beta pubsub subscriptions create mySubscription --topic myTopic \
   --push-endpoint=SERVICE-URL/ \
   --push-auth-service-account=pubsub-invoker@PROJECT-ID.iam.gserviceaccount.com

Add the service account email address pubsub-invoker@PROJECT-ID.iam.gserviceaccount.com to IAP for App Engine.

I don't know of a CLI command for this step. Do this step in the Google Cloud Console.

John Hanley
  • 74,467
  • 6
  • 95
  • 159
  • It worked !! Thank you. May I know what the above commands do ? – Jefila Oct 01 '19 at 01:36
  • I listed what each command does in my answer at both the top and for each command. – John Hanley Oct 01 '19 at 01:42
  • John. It worked yesterday when I tried pubsub with IAP enabled. But it is not working today. – Jefila Oct 01 '19 at 02:30
  • What is not working? You need to be specific. Check the Stackdriver logs for error messages. – John Hanley Oct 01 '19 at 02:34
  • I'm trying to publish a pubsub message from a microservice to another microservice, both of which are appengine projects. When pubsub message is published from 1st microservice, it should reach the push endpoint of the 2nd microservice. This functionality did work yesterday but today the pubsub messages did not reach the 2nd microservice endpoint. To facilitate this, I created a topic and subscription as mentioned by you in your answer. – Jefila Oct 01 '19 at 02:49
  • The microservice needs IAP enabled for the service to receive the Pub/Sub push. What service are you referring to? Your question is about App Engine. If you are using a different service create a new question. – John Hanley Oct 01 '19 at 03:03
  • Yes it is a microservice deployed in GCP app engine with IAP enabled for Pubsub push. – Jefila Oct 01 '19 at 03:10
  • Disable IAP for the service. Then re-enable making sure that you have the correct service account. Remember, these are beta features and not GA. – John Hanley Oct 01 '19 at 03:29
  • I tried disabling and re-enabling IAP, still not working. :( I wonder, why it worked yesterday and not today. – Jefila Oct 01 '19 at 03:33
  • Try redeploying the service. – John Hanley Oct 01 '19 at 04:30
  • Tried redeploying as well but no luck. May be Google should come up with some solution for this issue. – Jefila Oct 01 '19 at 09:31
  • The steps I provided are the Google solution. How is your microservice authenticating (providing the OAuth Identity Token) with the other service protected with IAP? – John Hanley Oct 01 '19 at 16:17
  • Each of my microservices communicate using IAP tokens with other Microservices protected my IAP. The direct REST call is working with this implementation. The issue I'm facing is with the Google pubsub Push for IAP enabled microservices. All my microservices are IAP enabled and residing in GCP Appengine. – Jefila Oct 02 '19 at 02:14
  • Hi John. Could you help me with any other solution as I find the above solution not working for my case .. :( – Jefila Oct 03 '19 at 08:22
  • I do this constantly in my projects. I have written articles on my website on how to use the identity features. If you can create a GitHub project with everything I need to reproduce your situation, I will look at it. – John Hanley Oct 03 '19 at 13:54
  • 2
    This solution worked for me, with a minor change. I had to set the Subscription's push authentication `Audience` field value to the IAP web app's OAuth Client ID. I found this ID here: https://console.cloud.google.com/apis/credentials – am9u Nov 26 '19 at 15:47
  • This solution did not solve the problem for me. Tried several times including adding audience, as one commenter advised, but messages are still not getting through. – Hippocrates Nov 28 '19 at 00:59
  • I found that if I add the pubsub invoker service account to the top-level App Engine resource rather than the individual service, the pubsub messages start flowing. It seems like a bug. – Hippocrates Dec 06 '19 at 20:00
  • @Hippocrates Can you tell me what is the top-level App engine resource you've mentioned? It is still not working for me. – Jefila Dec 09 '19 at 07:25
  • There will be a hierarchy in the IAP panel like so App Engine app > service-a, service-b, service-c Try adding the pubsub-invoker (setup described above) service account to that "App Engine app" resource instead of a specific service. It should have the "IAP-secured Web App User" role – Hippocrates Dec 09 '19 at 19:13
  • I tried to get this working also, even by adding the audience claim from the IAP configuration to the auth settings in the push subscriber. Also went as far as editing the iam policy to enable IAP audit logs. But the audit logs aren't helping me pinpoint the problem, they just confirm that there's a lot of permission denied messages coming into my IAP instance. – Michael Dec 11 '19 at 23:39
  • @Michael - Without details, I don't know. I would create a new question with your details. Let me know the link. – John Hanley Dec 12 '19 at 01:47
  • All good, I've got a case open with Google for now and will follow up here if there's something worth sharing. – Michael Dec 12 '19 at 21:45
  • Google support got back to me that there is an internal bug and they have ticketed it. The workaround I described above should work. – Hippocrates Dec 13 '19 at 21:02
  • @Hippocrates - Can you let me know the ticket number? – John Hanley Dec 13 '19 at 22:01
  • What I learned is that you need to take the client Id of the IAP and set that in the audience parameter for auth on the pubsub push subscription. A little odd considering IAP has an "audience" configuration parameter, but that fixed it for me. – Michael Dec 14 '19 at 23:56
  • https://issuetracker.google.com/146175841 To allow Pub/Sub to push to IAP-enabled app engine, service account must be added to top level App Engine level rather than service level The ticket ID is in the URL – Hippocrates Dec 15 '19 at 00:44
  • see here https://cloud.google.com/pubsub/docs/push#setting_up_for_push_authentication – Reza May 04 '21 at 18:37
  • note: this answer is from 2019, and now granting serviceAccountTokenCreator isn't needed. but the problem is it's not working - messages go to indelievered and GAE isn't called while if to disable IAP it starts working. – Shrike Mar 02 '22 at 01:15
  • Definitely a bug, I've raised an issue https://issuetracker.google.com/issues/264283872 – Rlweb Jan 04 '23 at 18:35