2

I have a running user-service managed Cloud Run Service. It is written in Kotlin and Spring Boot and I added the cloud logging library to it and added the logback.xml configuration neccessary.

A short prove:

    // build.gradle.kts
    
    implementation("com.google.cloud:google-cloud-logging-logback:0.119.4-alpha")

// logback.xml

<configuration>
    <springProfile name="!cloud, debug">
        <appender name="CONSOLE"
                  class="ch.qos.logback.core.ConsoleAppender">
            <layout class="ch.qos.logback.classic.PatternLayout">
                <Pattern>
                    %black(%d{ISO8601}) %highlight(%-5level) [%blue(%t)] %yellow(%C{1.}): %msg%n%throwable
                </Pattern>
            </layout>
        </appender>
        <root level="info">
            <appender-ref ref="CONSOLE" />
        </root>
    </springProfile>
    <springProfile name="cloud">
        <appender name="CLOUD" class="com.google.cloud.logging.logback.LoggingAppender"/>
        <root level="info">
            <appender-ref ref="CLOUD" />
        </root>
    </springProfile>
</configuration>

So I am running my application via Cloud Run with the SPRING_ACTIVE_PROFILES=cloud so the cloud logging part of the logback.xml should be active. It is - however it is throwing hundreds of exceptions

enter image description here

I have no idea what I did wrong. I followed this guide: https://cloud.google.com/logging/docs/setup/java

But there was nothing mentioned about any kind of authorization? So I can not explain the PERMISSION_DENIED issue here.

Here is the gcloud run services describe user-service output:

Traffic: https://user-service-53fsfabwe-ew.a.run.app
  100% LATEST (currently user-service-00051-xab)
Ingress: all
 
Last updated on 2021-01-23T10:08:05.204462Z by me@gmail.com:
  Revision user-service-00051-xab
  commit-sha:1bc273274cf191de6a4712d3f5b6f3cbafce42d2 gcb-build-id:07265ff6-f79b-4b1c-964a-41b4363856c2 gcb-trigger-id:8f88b2c2-eb93-4d3d-89a0-d841061f38c6 managed-by:gcp-cloud-build-deploy-cloud-run
  Image:             eu.gcr.io/mvp-prototype/user-service/user-service:1bc273274cf191de6a4712d3f5b6f3cbafce42d2
  Port:              8080
  Memory:            512Mi
  CPU:               1000m
  Service account:   user-service@mvp-prototype.iam.gserviceaccount.com
  Env vars:
    AUTH0_CLIENT_ID  <nope>
    AUTH0_CLIENT_SECRET <nope>
    AUTH0_DOMAIN     <nope>
    SPRING_PROFILES_ACTIVE cloud
    SQL_CONNECTION   10.28.96.3
    SQL_PASSWORD     test
    SQL_USER         test
  Concurrency:       80
  Max Instances:     1
  SQL connections:   mvp-prototype:europe-west1:prototype
  Timeout:           300s
  VPC connector:
    Name:            projects/mvp-prototype/locations/europe-west1/connectors/cloud-run-to-cloud-sql
    Egress:          private-ranges-only

Since the service-account is custom I checked if the proper IAM role for writing to logs is set. I therefore added the Logs Writer role to Service account: user-service@mvp-prototype.iam.gserviceaccount.com

However that did not help either. Still receiving the exception.

Here is the detailed log trace:

com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The caller does not have permission
    at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:201)
    at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:195)
    at com.google.api.core.ApiFutures$GaxFunctionToGuavaFunction.apply(ApiFutures.java:240)
    at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:224)
    at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:212)
    at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:124)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1215)
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
    at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)
    at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)
    at com.google.api.gax.rpc.BatchedFuture.setException(BatchedFuture.java:55)
    at com.google.api.gax.rpc.BatchedRequestIssuer.sendResult(BatchedRequestIssuer.java:84)
    at com.google.api.gax.rpc.BatchExecutor$1.onFailure(BatchExecutor.java:98)
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1041)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1215)
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
    at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:198)
    at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135)
    at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1215)
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
    at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)
    at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)
    at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1041)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1215)
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
    at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:563)
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The caller does not have permission
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
    ... 21 common frames omitted
Caused by: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The caller does not have permission
    at io.grpc.Status.asRuntimeException(Status.java:533)
    ... 13 common frames omitted
xetra11
  • 7,671
  • 14
  • 84
  • 159
  • Your service account still lacks some permissions I believe. Try giving it owner role and see if it helps. – Emil Gi Jan 25 '21 at 15:16
  • Hm I can try to give it the same permissions the default compute service account has. But I wonder why that is since the doc clearly says only `Logs Writer` is needed – xetra11 Jan 25 '21 at 15:18
  • Can you please link it? – Emil Gi Jan 25 '21 at 15:21
  • Also please sanitize sensitive data such as project id when posting questions. – Emil Gi Jan 25 '21 at 15:22
  • @EmilGi it's not my real project-id. It has been edited by me beforehand – xetra11 Jan 27 '21 at 14:42
  • Can you link the documentation you are referring to about permissions required? – Emil Gi Jan 27 '21 at 15:31
  • @EmilGi quickly I just found the python doc for it: https://cloud.google.com/logging/docs/setup/python – xetra11 Jan 27 '21 at 18:12
  • Anyway can you try to give it broader rol like owner and see if it works? Maybe some required permissions are not explicitly mentioned. Also make sure Cloud Logging API is enabled in the project. – Emil Gi Jan 28 '21 at 08:38
  • I set the role of the affected service's service-account to `Owner` now. Still the ` PERMISSION_DENIED: The caller does not have permission` exception occurs. I will now try to set it back to the default cloud-run service-account that has been set at initial creation of the run service – xetra11 Jan 28 '21 at 10:22
  • @EmilGi Ok after setting it back to the default service-account `991731337565-compute@developer.gserviceaccount.com` the mentioned exception is gone. The roles of that default service-account are `Cloud SQL Client, Editor`. So in fact it's not "higher" in hierarchy than the `Owner` role obviously. However I noticed one additional difference between the service-accounts. The "failing" account had this suffix `@mvp-prototype.iam.gserviceaccount.com` and the "non-failing" default one has `@developer.gserviceaccount.com`. Do those have any semantics for permissions? I thought it's just a namespace – xetra11 Jan 28 '21 at 10:33
  • Sorry for late reply, namespace difference should play any role and indeed `owner` role is higher in the hierarchy. Can you provide a simplified code sample so I could reproduce this issue? – Emil Gi Feb 02 '21 at 08:30

1 Answers1

0

added the role "Logging Admin" to the service account which fixed this issue in my case.

The error message was:- textPayload: "io.grpc.StatusRuntimeException: PERMISSION_DENIED: The caller does not have permission at io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.49.2.jar!/:1.49.2]

Nirmalya Kar
  • 511
  • 4
  • 9