Stack Overflow
Stack Overflow

Questions tagged [gcsfuse]

gcsfuse is a user-space file system for interacting with Google Cloud Storage.

149 questions
0
votes
1 answer

How can I achieve predicate pushdown when using PyArrow + Parquet + Google Cloud Storage?

What I'm really trying to do is this (in Python): import pyarrow.parquet as pq # Note the 'columns' predicate... table = pq.read_table('gs://my_bucket/my_blob.parquet', columns=['a', 'b', 'c']) First, I don't think that gs:// is supported in…
user5406764
  • 1,627
  • 2
  • 16
  • 23
0
votes
1 answer

Replace copy in finder with script using AppleScript

I have folder mounted using gcsfuse, and copying folders using finder is too slow, so i use gsutils in a bash script, is it possible to replace the default copy action in this folder using an AppleSctipt that runs my script? So if i drag a folder to…
José Pedro Machado
  • 1
0
votes
0 answers

Mount GCP Bucket in Linux with users' "own" access

I want to mount a GCP bucket on a ubuntu machine that allows user to upload files, and keep the file's ownership to the user. Currently, I have this line in my /etc/fstab: gcp-bucket-name /mnt/gcp-bucket-action gcsfuse…
Philip Shangguan
  • 449
  • 5
  • 19
0
votes
2 answers

gcsfuse terminating itself on GKE

I have gcsfuse in a deployment on GKE, it was working fine and without any changes in the config it started failing since yesterday. Received signal 15; terminating. I have it deployed in 2 different clusters, first I thought it was related to the…
Ivan Torrellas
  • 11
  • 2
0
votes
1 answer

Permission denied with gcsfuse in unprivileged Ubuntu-based Docker container

I was not able to run gcsfuse in my Ubuntu-based Docker image with --cap-add SYS_ADMIN --device /dev/fuse, as seen in other posts. It works like a charm with --privileged though, and with root or non-root user. But I would like to avoid this…
Pascal GILLET
  • 135
  • 1
  • 14
0
votes
1 answer

Getting this Error message when i added GSCFuse and deployed my application in Kubernetes Any Suggestions what could be the issue

poststarthookerror: command 'gcsfuse -o nonempty config_files_1_bucket /home/test123/' exited with 126 I am adding the gcs fuse command in my yaml file like lifecycle: postStart: exec: command: - gcsfuse -…
Lakshay Narang
  • 3
  • 1
0
votes
1 answer

GCP Cloud Composer - not able to download large file to data folder

Every week I have to download a file from ftp server to GCS bucket and then import that file from GCS bucket to BigQuery. I started implementing this dataflow job in GCP Cloud Composer. I broke dataflow into three tasks Download file from ftp…
Sri Harsha
  • 641
  • 2
  • 8
  • 18
0
votes
1 answer

Read/Write data to/from Cloud Storage Bucket using gcsfuse

To mount Google Cloud Storage Bucket onto a directory on a local machine for processing. Using a manjaro environment and installed gcsfuse manually. in the gs://bucket01, there are directories containing jpg and json files go get -u…
jj pan
  • 287
  • 2
  • 5
  • 17
0
votes
1 answer

Move large number of folders and files inside a GCS bucket

I have a bucket on GCP and at the top level of this bucket, I have a bunch of folders. I want to create a new folder and move all of the other ones into it. However, I've mounted my bucket with gcsfuse and tried traditional Linux mv commands. This…
rocksNwaves
  • 5,331
  • 4
  • 38
  • 77
0
votes
1 answer

Unknown bucket while doing gcsfuse

I want to fuse a gcs bucket with a local directory, for which I'm doing the following: gcsfuse gs://stereo-train ~/Documents/stereo-train However, I'm getting the error: Unknown bucket "gs://stereo-train" The stereo train bucket does exist on…
zendevil
  • 899
  • 4
  • 13
  • 26
0
votes
2 answers

Could not find default credentials google cloud

I want to fuse a local directory with a google cloud bucket with the following command: gcsfuse gs://stereo-train ~/Documents/stereo-train However, I'm getting the following error: Opening GCS connection... daemonize.Run: readFromProcess:…
zendevil
  • 899
  • 4
  • 13
  • 26
0
votes
1 answer

HDFS over S3 / Google storage bucket translation layer - how?

I'd love to expose a Google storage bucket over HDFS to a service. Service in question is a cluster (SOLR) that can speak only to HDFS, given I have no hadoop (nor need for it), ideally I'd like to have a docker container that would user a Google…
Jacek
  • 63
  • 5
0
votes
1 answer

How to configure gcsfuse to mount a bucket in a google cloud vm and access the resources in the best way?

When gcsfuse is used to mount a bucket in a google cloud vm and we access that resource, for example an image through a url that was configured for the vm, example.com/image.jpg, is internal traffic created between the vm instance and the bucket…
Gerson Rubio Garcia
  • 1
  • 1
  • 3
0
votes
1 answer

docker postgres not able to change pgdata permissions

I have a project where I use google compute engine to host my app and docker to containerize it. I have a postgres image and I want to use a volume to make my data persistent when I restart the container. Moreover I want the db data to be stored in…
Vetouz
  • 159
  • 3
  • 19
0
votes
0 answers

gcs fuse on two seperate instances

So I have two separate accounts I'm sharing a gcloud bucket between. At first I had problems getting the credentials right but eventually I just added all the email-looking accounts under the IAM on the second account to the storage buckets…
Zeinir
  • 25
  • 1
  • 5
1 2 3
…
9 10