Stack Overflow
Stack Overflow

Questions tagged [gcsfuse]

gcsfuse is a user-space file system for interacting with Google Cloud Storage.

149 questions
0
votes
1 answer

Accessing image data from google cloud buckets as a ML dataset

Please help if you can! I have a lot of individual images stored in a google bucket. I want to retrieve individual images from the bucket through google colab. I have already set up a connection via gcsfuse but I can still not access the images. I…
Joshua Clancy
  • 121
  • 1
  • 9
0
votes
1 answer

Cloud Composer/Airflow Task Runner Storage

I'm used to running pipelines via AWS data pipelines but getting familiar with Airflow (Cloud Composer). In data pipelines we would: Spawn a task runner, Bootstrap it, Do work, Kill the task runner. I just realized that my airflow runners are not…
JW2
  • 349
  • 2
  • 16
0
votes
1 answer

Is writing files to a gcloud storage bucket supposed to be slow?

I'm using a gcloud storage bucket mounted to a VM instance with gcsfuse. I have no problems opening files and reading them when the files are stored on the storage bucket, but when I try to write files to the storage bucket it is enormously slow…
logic1976
  • 519
  • 1
  • 6
  • 19
0
votes
2 answers

Not able to read files on gcsfuse mounted disk

I've been able to do this before so I know it works. In any case, I ran the command sudo gcsfuse deduction3 /mnt/disks/temp_dir Using mount point: /mnt/disks/temp_dir Opening GCS connection... Opening bucket... Mounting file system... File system…
logic1976
  • 519
  • 1
  • 6
  • 19
0
votes
1 answer

How to Use Cloud Storage Fuse to backup files when deleted in local machine

I installed gcsfuse on my local macOS system and mounted a folder to cloud storage bucket. everythings works fine. but, If deleted a file from mounted folder also deleting on bucket. I don't want this to be happen. when ever I delete any files, It…
Shashikumar KL
  • 1,007
  • 1
  • 10
  • 25
0
votes
1 answer

Can gsutil create the psudeo-directory entries needed to speed up gcsfuse?

Although Google Cloud Storage is a flat object store that doesn't need directory entries, adding psuedo directory placeholders (empty entries with names ending in /) makes gcsfuse a lot faster. You can leave out the gcsfuse --implicit-dirs option…
Jerry101
  • 12,157
  • 5
  • 44
  • 63
0
votes
1 answer

How to read from Google Cloud Storage CSV and Load it to Google Data Store

Can you please let me know how to read from Google Storage (CSV) file to Cloud Data Store. I had done, Java code via App Engine and can able to load one row at a invocation. Same way i would like to get some sample code to load bulk ( read from…
user12607246
  • 5
  • 3
0
votes
1 answer

ModuleNotFoundError: No module named 'fuse'

I have set up a GPU Jupyter Notebook VM using the AI platform on Google Cloud. The server runs Debian stretch. I want to mount a bucket I've created called example onto a folder called /home/jupyter/transfer. I've been following the instructions…
andmck
  • 101
  • 4
0
votes
1 answer

How to increase gcsfuse throughput when looping through many files?

I'm processing over 200,000 netcdf files and each file is 17 MB. They are all in a google cloud storage bucket and I am trying to find a way to increase the throughput using gcsfuse. I am using the google cloud compute engine virtual machine and…
yombob
  • 59
  • 6
0
votes
1 answer

GCP Instance Metadata not passing scope

I am trying to mount a GCP storage bucket using gcsfuse and service account authentication. Everything mount OK but i only have read permission, see below; hannel@gcpxxxxxxd1:~$ gcsfuse --dir-mode 777 --file-mode 777 test /testing Using mount point:…
Hannel
  • 1,656
  • 3
  • 10
  • 17
0
votes
1 answer

Using gcsfuse during image build phase

I'm trying to use gcsfuse during docker build ... command in order to mount a working directory into a GCP storage, I've asked the question in Server Fault as the gcsfuse support is redirected to it but I can't find a way to make it works and as…
Guikingone
  • 21
  • 1
  • 5
0
votes
1 answer

mounting bucket with fstab not working NEWBIE

I'm new on GCP and on linux and I try to mount a bucket on my centos instance using gcsfuse. I tried with a script running at boot but it was not working so I tried with fstab (peoples told me it is much better) But I got this error when I tried to…
Anthony Fasano
  • 71
  • 11
0
votes
2 answers

OSError input/output error when writing to mounted filesystem

I have created Google Cloud Storage Bucket and mounted it to my VM using: gcsfuse my-bucket /path/to/mount/point When I type df -H its listed as mounted filesystem. I have the following issue: If I upload file to the bucket using the GCP web…
John
  • 329
  • 1
  • 20
0
votes
1 answer

How to run container with privileged flags using KubernetesPodOperator in GoogleCloudComposer?

I need to use a container with gcsfuse in google cloud composer, as far as i know --privileged flags when running the container is needed for gcsfuse, how to use --privileged flags when running the container using kubernetes pod operator in google…
Rahandi Noor Pasha
  • 41
  • 8
0
votes
1 answer

Access random line in large file on Google Cloud Storage

I'm trying to read a random line out of a large file stored in a public cloud storage bucket. My understanding is that I can't do this with gsutil and have looked into FUSE but am not sure it will fill my use…
user491880
  • 4,709
  • 4
  • 28
  • 49
1 2 3
…
9
10