Why does Request Sync on HomeGraph API return 403 Forbidden?

Question

Problem

When I call "Request Sync" on the Google HomeGraph API I receive a "403 Forbidden" response.

Background

I'm writing a Smart Home Action, and have successfully implemented SYNC, QUERY and EXECUTE. Testing on my mobile I can see and interact with devices okay. I'm now trying to implement Request Sync, but can't appear to interact with the API. I am making what seems to be successful requests for an Access Token. The token always begins with "ya29.c." which in my naïve understanding suggests an empty header and payload (trying it on https://jwt.io). However, when testing it at https://accounts.google.com/o/oauth2/tokeninfo?access_token= it appears valid, showing both my service account unique ID and the scope I intended. When I make a call to the API, either manually posting the data, or via Google's own code, it gives me a blunt 403 error. I do not know where I can get any more information on this error other than the exception objects. I'm new to GCP and couldn't find any sort of log. Given I've tried different methods and all return a 403 I'm inclined to suspect the issue is more a account or credential-related than the code, but can't be certain.

API Key

(I'm no longer able to reproduce any errors relating to API keys being missing or invalid).

Although the documentation doesn't show it, I've seen some people use an API key. When I don't include the API key with a p12 certificate, or include an incorrect one it errors (either with API key missing, or API key invalid accordingly). I have created an unrestricted API key in IAM, and am using that. I can't appear to explicitly relate this to HomeGraph API, but it says that it can call any API.

Code

This example fetches an access token, then tries to call the API via POST with and without the API key. It then tries to authenticate and call the API via the Google library code. Each fails with a 403.

using Google;
using Google.Apis.Auth.OAuth2;
using Google.Apis.HomeGraphService.v1;
using Google.Apis.HomeGraphService.v1.Data;
using Google.Apis.Services;
using Lambda.Core.Constants;
using System.IO;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using static Google.Apis.HomeGraphService.v1.DevicesResource;

public class Example
{
    public void RequestSync()
    {
        const string UrlWithoutKey = @"https://homegraph.googleapis.com/v1/devices:requestSync";
        const string UrlWithKey = @"https://homegraph.googleapis.com/v1/devices:requestSync?key=" + OAuthConstants.GoogleApiKey;
        string accessToken = this.GetAccessToken();

        // Manual Attempt 1
        try
        {
            string response = this.CallRequestSyncApiManually(accessToken, UrlWithoutKey);
        }
        catch (WebException ex)
        {
            // Receive 403, Forbidden
            string msg = ex.Message;
        }

        // Manual Attempt 2
        try
        {
            string response = this.CallRequestSyncApiManually(accessToken, UrlWithKey);
        }
        catch (WebException ex)
        {
            // Receive 403, Forbidden
            string msg = ex.Message;
        }

        // SDK Attempt
        try
        {
            this.CallRequestSyncApiWithSdk();
        }
        catch (GoogleApiException ex)
        {
            // Google.Apis.Requests.RequestError
            // The caller does not have permission[403]
            // Errors[Message[The caller does not have permission] Location[- ] Reason[forbidden] Domain[global]]
            //  at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response) in Src\Support\Google.Apis\Requests\ClientServiceRequest.cs:line 243
            //  at Google.Apis.Requests.ClientServiceRequest`1.Execute() in Src\Support\Google.Apis\Requests\ClientServiceRequest.cs:line 167
            string msg = ex.Message;
        }
    }

    private string GetAccessToken()
    {
        string defaultScope = "https://www.googleapis.com/auth/homegraph";
        string serviceAccount = OAuthConstants.GoogleServiceAccountEmail; // "??????@??????.iam.gserviceaccount.com"
        string certificateFile = OAuthConstants.CertificateFileName; // "??????.p12"
        var oAuth2 = new GoogleOAuth2(defaultScope, serviceAccount, certificateFile); // As per https://stackoverflow.com/questions/26478694/how-to-produce-jwt-with-google-oauth2-compatible-algorithm-rsa-sha-256-using-sys
        bool status = oAuth2.RequestAccessTokenAsync().Result;

        // This access token at a glance appears invalid due to an empty header and payload,
        // But verifies ok when tested here: https://accounts.google.com/o/oauth2/tokeninfo?access_token=
        return oAuth2.AccessToken;
    }

    private string CallRequestSyncApiManually(string accessToken, string url)
    {
        string apiRequestBody = @"{""agentUserId"": """ + OAuthConstants.TestAgentUserId + @"""}";
        var client = new HttpClient();
        var request = (HttpWebRequest)WebRequest.Create(url);
        var data = Encoding.ASCII.GetBytes(apiRequestBody);
        request.Method = "POST";
        request.Accept = "application/json";
        request.ContentType = "application/json";
        request.ContentLength = data.Length;
        request.Headers.Add("Authorization", $"Bearer {accessToken}");
        client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);

        using (var stream = request.GetRequestStream())
        {
            stream.Write(data, 0, data.Length);
        }

        var response = (HttpWebResponse)request.GetResponse();
        var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();

        return responseString;
    }

    private void CallRequestSyncApiWithSdk()
    {
        var certificate = new X509Certificate2(OAuthConstants.CertificateFileName, OAuthConstants.CertSecret, X509KeyStorageFlags.Exportable);

        var credential = new ServiceAccountCredential(
           new ServiceAccountCredential.Initializer(OAuthConstants.GoogleServiceAccountEmail)
           {
                   Scopes = new[] { "https://www.googleapis.com/auth/homegraph" },
           }.FromCertificate(certificate));

        var service = new HomeGraphServiceService(
            new BaseClientService.Initializer()
            {
                // Complains if API key is not provided, even though we're using a certificate from a Service Account
                ApiKey = OAuthConstants.GoogleApiKey,
                HttpClientInitializer = credential,
                ApplicationName = OAuthConstants.ApplicationName,
            });

        var request = new RequestSyncRequest(
            service,
            new RequestSyncDevicesRequest
            {
                AgentUserId = OAuthConstants.TestAgentUserId
            });

        request.Execute();
    }
}

Account Configuration

Account screenshots. (I'm not allowed to post images yet, so they're links)

HomeGraph is enabled

My API Key is unrestricted

My Service Account has Owner & Service Account Token Creator enabled

Updates

I have tried skipping manually obtaining the access token, as per Devunwired's suggestion. Whilst this does eliminate the error I was getting from not providing the API key, I still end up with the 403. My reasoning for doing the access token part manually was part of debugging a 403 I was getting with the API call. That way I could at least see part of the process working. I'm happy to use the library version for the solution as the access token doesn't appear to be the issue.

public void GoogleLibraryJsonCredentialExample()
{
    try
    {
        GoogleCredential credential;

        using (var stream = new FileStream(OAuthConstants.JsonCredentialsFileName, FileMode.Open, FileAccess.Read))
        {
            credential = GoogleCredential.FromStream(stream).CreateScoped(new[] { OAuthConstants.GoogleScope });
        }

        var service = new HomeGraphServiceService(
            new BaseClientService.Initializer()
            {
                HttpClientInitializer = credential,
                ApplicationName = OAuthConstants.ApplicationName,
            });

        var request = new RequestSyncRequest(
            service,
            new RequestSyncDevicesRequest
            {
                AgentUserId = OAuthConstants.TestAgentUserId
            });

        request.Execute();
    }
    catch (Exception ex)
    {
        // Receive 403, Forbidden
        string msg = ex.Message;
    }
}

Concerns

Is it possible that I need to be making the API call from a verified or white-listed domain? At the moment I'm running it from a console app running on my development machine. My understanding of domain verification is that it does not apply to incoming calls, and therefore shouldn't be the problem.

Answer 1

I am making what seems to be successful requests for an Access Token.

You should not need to manually request OAuth access tokens when using the Google client libraries. They generally handle this process internally using the credentials you provide from the GCP console.

Although the documentation doesn't show it, I've seen some people use an API key. Indeed, it is mandatory to include it for the SDK approach.

We do not recommend using the API key method to access the Home Graph API. You should be using service account credentials. API keys will technically work for the Request Sync method, but you will not be able to authenticate Report State using an API key.

The fact that you are receiving an error trying to build the HomeGraphServiceService without an API key may be indicative that the credential you are using isn't set up correctly (no private key or possibly missing scopes). The recommended method for supplying service account credentials is to download them in the JSON format rather than certificate, and the code to generate a credential from JSON should look something like this:

GoogleCredential credential;
using (var stream = new FileStream(serviceAccountCredentialFilePath, FileMode.Open, FileAccess.Read))
{
    credential = GoogleCredential.FromStream(stream).CreateScoped(scopes);
}

You can find additional C# examples for authenticating APIs in the authentication guide.

Answer 2

The problem was nothing to do with my permission to talk to the HomeGraph API or that user. Instead it was where HomeGraph wanted to call my Smart Home Action, but the access token had expired. When attempting to refresh the token, an erroneous implementation on my part led to a blunt 403, which Google was then relaying back to me.

For those interested, the issue was that rather than omitting the expiry date for a token that should never expire, I was setting it to DateTime.MaxValue (subsequently sent through some further processing). Unfortunately when this is finally cast to an int, it is a value that exceeds int.Max. The subsequent time on the expiry was set to epoch (i.e. in the past), and therefore the token validation failed due to expiry.

For anyone else still having the same issue, double check your agentUserId matches exactly the value shown in your SYNC output payload. In my case I'd checked this.

Many thanks to anyone who's looked at this.