Stack Overflow
Stack Overflow

Questions tagged [gcsfuse]

gcsfuse is a user-space file system for interacting with Google Cloud Storage.

149 questions
6
votes
1 answer

gcsfuse to mount a bucket in GKE and/or python3 boto to stream write?

I am looking for a way to "write stream" some .mp4 video files -- as they are being generated by some python app -- to a google cloud storage bucket. The python app is containerised and deployed in GKE and currently executes fine as a web service.…
Aida
  • 2,174
  • 2
  • 16
  • 33
6
votes
3 answers

use gcsfuse to mount google cloud storage buckets in a docker container

I am trying to mount a google cloud bucket from within a docker container and get the following error: [root@cdbdc9ccee5b workdir]# gcsfuse -o allow_other --debug_gcs --key-file=/src/gcloud_service_account.json my-bucket-name /gcloud Using mount…
Marshall
  • 413
  • 1
  • 5
  • 10
6
votes
3 answers

Mounting google storage bucket folder using gcsfuse

I am not able to mount a specific folder inside the google storage bucket On OS X, this program allows you to mount buckets using the mount command. (On Linux, only root can do this.) mount -t gcsfuse -o rw,user my-bucket /path/to/mount/point On…
user28763
  • 111
  • 1
  • 2
  • 4
5
votes
1 answer

Unable to mount bucket with gcsfuse on Cloud Run

With the second generation runtime of Google Cloud Run, it's now possible to mount Google Storage Buckets using gcsfuse. https://cloud.google.com/run/docs/tutorials/network-filesystems-fuse The python3 example is working fine. Unfortunately, I keep…
Fred
  • 868
  • 10
  • 22
4
votes
4 answers

Can I use Cloud Shell with more than the 5 GB persistent storage?

According to the docs: Cloud Shell provisions 5 GB of free persistent disk storage mounted as your $HOME directory on the virtual machine instance. I would need more (paid) storage though that I can access from the Cloud Shell environment and that…
Ani
  • 1,377
  • 1
  • 18
  • 29
4
votes
4 answers

Running Docker on Google Cloud Instance with data in gcsfuse-mounted Bucket

I am trying to run a Docker container to analyze data in a Google Cloud Bucket. I have been able to successfully mount the Bucket using gcsfuse, and I tested that I could do things like create and delete files within the Bucket. In order to be able…
Charles W
  • 53
  • 7
4
votes
1 answer

gcsfuse on datalab vm machine. Error: fusermount: fuse device not found, try 'modprobe fuse' first

I have installed gcsfuse on a datalab machine. Created a target machine and used chmod to allow writing permission to all calling: !gcsfuse --foreground --debug_fuse archs4 /content/datalab/mount/ I am getting the following error: Opening…
eilalan
  • 669
  • 4
  • 20
4
votes
2 answers

What is the best equivalent of a file server on google compute engine

We are trying to move from the datacenter to the Google Compute Engine. While we understand how we can setup instances, and deploy the workload, we are not sure what is the best equivalent of storing data. We receive data once every day, and there…
Humble Debugger
  • 4,439
  • 11
  • 39
  • 56
3
votes
1 answer

Cloud storage directory disappears when restarting instance

I have attached a standard persistent disk to the instance. On that disk I have created a directory 'cloud-storage' (as always mkdir) in which I mount the bucket (Google Cloud Storage). Also I have added to fstab the command to mount it: bucket_name…
Apyc
  • 307
  • 5
  • 12
3
votes
1 answer

google colab with google cloud storage data egress

I'm trying to teach my network on data that is stored in google cloud storage. I'm teaching using google colab pro resources and when I do that, I got around 50$ bill a day for "egress between NA and EU". I'm located in Russia and data storage is…
Saffer
  • 152
  • 8
3
votes
2 answers

How to mount GCP Bucket in Google Colab

I want to access the files in my GCP bucket from Colab. I followed these instructions As you can see from the screen shots, there is nothing in the folder after mounting. What am I missing? The Data folder in my bucket is full of data.
jjschuh
  • 187
  • 1
  • 2
  • 8
3
votes
1 answer

gcsfuse won't write to folder

I am trying to figure out why if I don't need to do this step (Ubuntu before wily only) Add yourself to the fuse group, then log out and back in: sudo usermod -a -G fuse $USER exit Then why can't I write to files; I keep getting the following…
RussellHarrower
  • 6,470
  • 21
  • 102
  • 204
3
votes
0 answers

gcsfuse gives Input/Output error when reading

I have mounted a bucket with: hostname:/ # gcsfuse --implicit-dirs --foreground bucketname /directory Using mount point: /directory Opening GCS connection... WARNING: gcsfuse invoked as root. This will cause all files to be owned by root. If this is…
Envite
  • 321
  • 2
  • 6
  • 16
3
votes
0 answers

Apache beam python error handling while reading files from file pattern

I am reading all files from a directory and writing to the bigquery table. If there is an error with any file in the directory, it will raise the error and stop the job. I am not getting any information about that file (file name where error has…
Shravani G
  • 31
  • 1
3
votes
0 answers

Variable Performance when Training Resnet on Google Compute Engine with data from Google Cloud Storage

I'm attempting to train a resnet18 image classifier on a large number of crops from labeled Google Streeview data. I'm following along with this tutorial. I have two datasets, one of approximately ~20k images and one of approximately ~100k images.…
galenweld
  • 31
  • 3
1
2
3
…
9 10