GKE 1.4.5 (master and nodes) on gci cannot access cloud storage

Question

I am attempting to migrate our GKE cluster (running 1.4.5) from container-vm to gci. I am using the migration guide at https://cloud.google.com/container-engine/docs/node-image-migration

When I get my deployment on a gci node, I cannot access cloud storage, I use the google-cloud java 0.5.0 library (same issue with 0.4.0); the error I get is an insufficient rights, there are no problems on the container-vm node.

Here are my cluster permissions :

User info                  Enabled
Compute                    Read Write
Storage                    Read Write
Task queue                 Enabled
BigQuery                   Enabled
Cloud SQL                  Enabled
Cloud Datastore            Enabled
Stackdriver Logging API    Full
Stackdriver Monitoring API Full
Cloud Platform            Enabled
Bigtable Data             Read Write
Bigtable Admin            Full
Cloud Pub/Sub             Enabled
Service Control           Enabled
Service Management        Read Write
Stackdriver Trace         Disabled
Cloud Source Repositories Disabled

here is the error I get:

com.google.cloud.storage.StorageException: Insufficient Permission
at com.google.cloud.storage.spi.DefaultStorageRpc.translate(DefaultStorageRpc.java:202) ~[google-cloud-storage-0.5.0.jar:0.5.0]
at com.google.cloud.storage.spi.DefaultStorageRpc.create(DefaultStorageRpc.java:253) ~[google-cloud-storage-0.5.0.jar:0.5.0]
...
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
    {
       "code" : 403,
       "errors" : [ {
         "domain" : "global",
         "message" : "Insufficient Permission",
         "reason" : "insufficientPermissions"
       } ],
       "message" : "Insufficient Permission"
    }

How are you authenticating the request in your code, using default GCE service account or another service account in your project? Also can you confirm if your service account has the appropriate permissions(editor) setup in the project? — Faizan, Nov 16 '16 at 00:46
yes, everything is authenticated, with proper permission, again this does not happen with the container-vm image just the GCI image. I received an answer from GCP support folks saying that somehow the permissions were not getting populated on the GCI vm and that I should do an in-place VM update. Given the risks involved with the in-place update, I decided to opt-out and stay on container-vm for this cluster. — user1568967, Nov 16 '16 at 11:55

Faizan · Accepted Answer · 2016-11-24T18:34:08.947

I believe you have created the new node-pool through web UI(cloud console). If so, this is a known UI issue where you don't have an option to specify the scopes for the new node pool. Engineering team is already aware of this limitation and is working on a fix. With that said, if the scopes are not specified the default scopes are used for the new node pool. The workaround is either to upgrade the cluster using gcloud upgrade command with the gci image type or you can use gcloud to create the node pools and provide the necessary scopes for them as mentioned here.

GKE 1.4.5 (master and nodes) on gci cannot access cloud storage

1 Answers1