Stack Overflow
Stack Overflow

Questions tagged [gcs]

237 questions
0
votes
2 answers

what is the best way to read US region GCS bucket to write the data into Europe region bucket

The need is to read GCS bucket(bucket 1) mapped to US region and then write into Europe region GCS bucket(bucket 2). I would like to find the best solution to achieve the requirement. I thought to implement bucket-1 as multiregional but I could see…
Sekhar
  • 627
  • 4
  • 14
  • 34
0
votes
0 answers

What are the correct permissions to give Argo workflows given googleapi: Error 403?

I am seeing the following error running Argo workflows in GKE. time="2022-05-13T14:17:40.740Z" level=info msg="node changed" new.message="Error (exit code 1): upload /tmp/argo/outputs/artifacts/message.tgz: writer close: googleapi: Error 403: Access…
user3877654
  • 1,045
  • 1
  • 16
  • 40
0
votes
1 answer

Copy from S3 to GCS files with different date

I have to move files from S3 to GCS. The problem i have is on mondays they uploads files from monday but also of saturdays and sundays and this files have different dates. For example: stack_20220430.csv, stack_20220501.csv. I need to move this…
Gon Alb
  • 41
  • 6
0
votes
1 answer

Create google storage bucket using terraform and set life cycle rule for the specified buckets

I have a terraform module to create some storage buckets with specified permissions to users now I need to specify some life cycle rules for some of the buckets, and I need to map between two variables as follows: variable.tf variable "buckets" { …
Mohammad Varmazyar
  • 1
  • 1
0
votes
0 answers

Rename an object in Google cloud storage

I am trying to rename the cloud storage blob name but after running cloud function there is no change in blob name. Please find below code: from google.cloud import storage def rename_blob(bucket_name, blob_name, new_name): """Renames a…
shivam
  • 111
  • 2
  • 13
0
votes
1 answer

What is the GCS base PATH?

D'oh! I was trying to add a directory to my PATH and made a typo that deleted the base path. I still have access to my Google Cloud Shell, but I do not have a list of the directories in the base PATH. Can anyone tell me what that would be so that I…
user3877654
  • 1,045
  • 1
  • 16
  • 40
0
votes
1 answer

Unzipping files with Apache Beam (Python) but when using WriteToText it puts all columns as lines

I am very new to programming and Apache Beam, and I am trying to read plenty zip files on a a GCS bucket and unzip them and save again as csv on GCS. with beam.Pipeline() as pipeline: readable_files = ( pipeline |…
Eveline Marques Guimarães
  • 1
0
votes
3 answers

Is there a way to provide schema or auto-detect schema when uploading csv from GCS to BigQuery?

I am trying to upload a csv file from Google Cloud Storage (GCS) to BigQuery (BQ) and auto-detect schema. What I tried to do is enable auto-detect schema and enter the number of rows to skip in "Header rows to skip" option. I have 6 rows which…
R.A
  • 97
  • 1
  • 12
0
votes
0 answers

GCS bucket generating presigned url for uploads

I am trying to generate the presigned url for uploading files to buckets in GCS. But am having the following error when trying to open the link in the browser. Following is my code package test; import com.google.auth.Credentials; import…
user846445
  • 221
  • 1
  • 3
  • 14
0
votes
1 answer

How to keep generation number when copy file from gcs bucket to another bucket

I'm using gcs bucket for wordpress (wp-stateless plugin) after create and upload media file to a bucket. I copy it to other bucket (duplicate). But generation number of each object has been change (maybe random). My question is: How to keep…
Trần Văn Kiên
  • 32
  • 7
0
votes
1 answer

google sheet upload to GCS button

I have to create a scripted button in my google sheet to upload it as CSV into GCS(buckets). I have found a nice example documentation but i see there is API keys in it: https://www.labnol.org/code/20074-upload-files-to-google-cloud-storage I would…
Jonito
  • 407
  • 7
  • 18
0
votes
1 answer

GCP rename files with same name as existing directory

I'm trying to rename my file toto.html by removing its extension using gsutils. My folder/files are organized this way: toto ----| | |--file1.html | |--file2.txt | |--…
zoey3
  • 170
  • 1
  • 2
  • 14
0
votes
1 answer

Python script to count the number of records in a file on Google cloud bucket

Can you please help on providing python script to capture count of records in a file that is on GCS. Im trying to connect from linux server to GCS Bucket and capture the count of records/size of file.
Prashanth
  • 109
  • 1
  • 5
0
votes
0 answers

Why can't I catch GCS excpetions

import com.google.cloud.storage.Blob; import com.google.cloud.storage.StorageException; import com.google.cloud.storage.StorageOptions; public class CleanUp { public static void main(String... args) { final var storage =…
Eric Kolotyluk
  • 1,958
  • 2
  • 21
  • 30
0
votes
1 answer

execute gsutil from airflow using python callable

we want to run "gsutil ls -L gs://bucket/obj1" command to print the information of gcs object. we are using airflow to connect with gcs. can anyone help with syntax to run above command using python callable function
Yug
  • 105
  • 1
  • 9
1 2 3
…
15 16