Unable to write files in a GCP bucket using gcsfuse

Question

I have mounted a storage bucket on a VM using the command:

gcsfuse my-bucket /path/to/mount

After this I'm able to read files from the bucket in Python using Pandas, but I'm not able to write files nor create new folders. I have tried with Python and from the terminal using sudo but get the same error.

I have also tried Using the key_file from the bucket:

sudo mount -t gcsfuse -o implicit_dirs,allow_other,uid=1000,gid=1000,key_file=Notebooks/xxxxxxxxxxxxxx10b3464a1aa9.json <BUCKET> <PATH>

It does not through errors when I run the code, but still I'm not able to write in the bucket.

I have also tried:

gcloud auth login

But still have the same issue.

Add the approach you're using to read/write the files as well — Abhinav Mathur, Sep 18 '22 at 16:18
You have to work with custom access scopes on the virtual machine. By default this is read-only, so that is why you are seeing this behaviour. — Cloudkollektiv, Sep 18 '22 at 19:33

Cloudkollektiv · Accepted Answer · 2022-09-18T19:20:29.150

I ran into the same thing a while ago, which was really confusing. You have to set the correct access scope for the virtual machine so that anyone using the VM is able to call the storage API. The documentation shows that the default access scope for storage on a VM is read-only:

When you create a new Compute Engine instance, it is automatically configured with the following access scopes:

Read-only access to Cloud Storage: https://www.googleapis.com/auth/devstorage.read_only

All you have to do is change this scope so that you are also able to write to storage buckets from the VM. You can find an overview of different scopes here. To apply the new scope to your VM, you have to first shut it down. Then from your local machine execute the following command:

gcloud compute instances set-scopes INSTANCE_NAME \
  --scopes=storage-rw \
  --zone=ZONE

You can do the same thing from the portal if you go to the settings of your VM, scroll all the way down, and choose "Set Access for each API". You have the same options when you create the VM for the first time. Below is an example of how you would do this:

Unable to write files in a GCP bucket using gcsfuse

1 Answers1