Stack Overflow
Stack Overflow

Questions tagged [gcs]

237 questions
1
vote
0 answers

Unable to access uploaded file (go + testcontainers + fsouza/fake-gcs-server)

Here is an issue with the fsouza/fake-gcs-server emulator - I use testcontainers for integration test in go - uploading works fine (I see the file in the container) but there is no way to access it from the code (to assert it was indeed uploaded),…
Robert Trzebiński
  • 1,327
  • 8
  • 16
1
vote
0 answers

I Always Got Cors Error When Send DELETE request to gcs Resumable Singed Url

I've tried to use "resumable signed url" when uploading a file. This is the CORS config on the bucket: [{“maxAgeSeconds”: 3600, “method”: [“GET”, “PATCH”, “DELETE”, “OPTIONS”, “POST”, “PUT”], “origin”: [“*”], “responseHeader”: [“*”]}] BE const…
jiyoung
  • 11
  • 1
1
vote
2 answers

GCS and Java- Concatenate dynamic bucket name and file name in TestIO

I want to write a file to a GCS bucket. The bucket path and file name are dynamically provided in two different pipeline options. How can I concatenate those in TextIO to write the file to the GCS bucket. I tried doing this but no…
pas
  • 109
  • 1
  • 13
1
vote
1 answer

1 column with different schemas across files in folder in Spark (PlainDoubleDictionary)

TLDR Data Source: GCS target BigQuery Problem wildcard reads multiple files, all with same columns, but airport_fee sometimes is of integer and sometimes of double datatype Error: java.lang.UnsupportedOperationException:…
Islam Elsayed
  • 21
  • 1
  • 4
1
vote
1 answer

Error posting object to GCS with Signed Post policy

I'm trying to create a signed policy for users to upload their files on Google Cloud Storage. The Issue i'm facing here is with the filename, i want the user to provide the filename at the time of the upload, as mentioned in the official GCS…
Husain Batatawala
  • 37
  • 1
  • 6
1
vote
0 answers

Adding "missing" columns to a CSV file so it can be correctly loaded into a BigQuery external table

I have dozens of CSV files that start in an S3 bucket, and then are transferred to a GCS bucket using the GCP storage transfer service. I can't do anything about the source files in S3. Some files have 30 columns, while others will have fewer (as…
Sultan of Swing
  • 430
  • 1
  • 6
  • 20
1
vote
0 answers

Copying files from sharepoint to google cloud sotrage based on event trigger

i need to copy files from a sharepoint to GCS automatically: everytime there is a new file in the sharepoint,it should be copied automatically to GCS? any hint please ?
user19190095
  • 11
  • 1
1
vote
1 answer

How to pass URIs as variable in BQ stored procedure

I am new to BQ and trying to load table via file from GCS bucket and getting error for uris . Query error: Found unsupported function call 'ARRAY[...]'; failed to set 'uris' in OPTIONS() Code Snippet : begin declare filename STRING; declare…
Mudgal
  • 35
  • 4
1
vote
1 answer

copy a list of files using gsutil from one gcs bucket to another

I have a thousands of files in one gcs bucket. Out of which i wanted to copy some n list of files using gsutil -m cp command. By reading the documentation I can perform something like this in my python script. cat filelist | gsutil -m cp -I…
ramkrishs
  • 78
  • 7
1
vote
2 answers

Google Cloud Storage To Google Cloud SQL (Postgres) Operator in Airflow (or Composer)

I am trying to load data from a CSV file in GCS, but there is no predefined operator that does this in Airflow. I built a simple operator using a PSQL hook and a GCS file reader, but I'm wondering if there is a better solution for this, as right now…
johan855
  • 1,578
  • 4
  • 26
  • 51
1
vote
0 answers

Loki-stack Helm Chart Storage and Retention with GCS

I'm running a local kubernetes cluster (minikube) and I have grafana installed in a namespace and the loki-stack helm chart installed in another namespace. I can query logs on my grafana instance and everything works out of the box. (I'm using the…
skarsky
  • 63
  • 1
  • 6
1
vote
2 answers

How to resolve TypeError: __init__() got an unexpected keyword argument 'page_size' while using google cloud storage

I have been trying to read from gcs with the following code: from google.cloud import storage client = storage.Client() len(list(client.list_blobs('mybucket', prefix='dir1/dir2'))) however i get the following error message: TypeError …
Mim
  • 33
  • 5
1
vote
2 answers

After switching google cloud projects my python cannot find my new google project's storage bucket?

I created a new project and even though all the IAM permissions are set and local login credentials are correct, I cannot connect to my bucket using python's google storage API (Windows) I updated the gcloud authentication and it shows I am using…
Anthony M
  • 109
  • 8
1
vote
1 answer

Uncompress file in GCS with Python - How to add project and auth

In this code from google.cloud import storage from zipfile import ZipFile from zipfile import is_zipfile import io def zipextract(bucketname, zipfilename_with_path): storage_client = storage.Client() bucket =…
Elesh Mistry
  • 11
  • 4
1
vote
0 answers

Is my use case for GCP Dataproc feasible?

Not sure if there is a place/people to ask for one on one advice for Dataproc setup and tuning. But figure here is as good as place as any to find some help. Our team has been primarily using BigQuery to do our data analysis on location driven data.…
cpatino08
  • 95
  • 2
1 2
3
…
15 16