Stack Overflow
Stack Overflow

Questions tagged [gcs]

237 questions
0
votes
1 answer

Load huge data to client side

We have APIs written in the back end(Flask) and are being consumed by front end (React JS). In the back end, we are merging CSV files using pandas, so after merging all the CSVs we have around 1 million records that we need to give to the FE to…
Abhishek Pore
  • 1
0
votes
0 answers

Handle full snapshot data coming as part files in data pipeline in GCP

I am working on writing data pipelines for our data lake using GCP. We have a requirement of getting a full snapshot for one of our data sources. Since the snapshot is big in size, the ingestion spark system creates multiple part files and copies…
Harbeer Kadian
  • 364
  • 2
  • 14
0
votes
0 answers

Issue with gcs upload of an image file

I'm facing an issue in uploading image file, so far I've seen it with png files, into a gcs buckets specific folder. The env logs indicate that it has been uploaded successfully but the file is nowhere to be found. There is no TTL and I've also…
Mentee
  • 1
0
votes
0 answers

REST API - GCS file in Payload request

I am trying to upload a file to POST API request. I am able to give the file as input to the request if it is my local system path, but when I productionize this, the file should be taken from GCS bucket. Any inputs on this? Below is the code used…
Sri Bharath
  • 115
  • 1
  • 2
  • 10
0
votes
1 answer

How Class A operations is calculated in GCP

I have tried replicating this but it seems bit confusing still. Scenario: I have 10 Hive tables, each pointing to different location within same GCS bucket. For example, table1 points to gs://bucket/table1, table2 points to gs://bucket/table2 and so…
dipayan
  • 128
  • 5
  • 18
0
votes
0 answers

Is there a way to avoid a gsutil copy from cloud to local machine throwing a 404 midway through download?

Running gsutil cp ${gsutilURI} ${localPath} will sometimes return a 404 midway through download. Retrying the command will eventually succeed. In the script running the above command, there is a prior existence check with gsutil -q stat ${gsutilURI}…
Alex G
  • 1
  • 2
0
votes
0 answers

Writing Generic Record as Avro Files to GCS Bucket in Java

I am trying to write generic records to a GCS bucket as avro, On writing I can observe two things The File type in GCS is application/octet-stream instead of avro Some of the data are missing i.e are null, especially the nested fields Here is the…
jeks
  • 3
  • 2
0
votes
0 answers

cloud bucket blob download is very slow in cloud run

I am using python sdk to download large (1Gb) object from cloud storage bucket inside cloud run object. cloud run has 8 GB memory and 4 cpus. I tried it with various chunk size and various worker count. Below is my code: bucket =…
Djai
  • 188
  • 10
0
votes
1 answer

FileNotFoundError using gcsfs and pandas, but only on my machine

I have used pandas and gcsfs regularly in the past. Recently, I have been getting errors when trying to do so. I cannot reproduce the error on other systems. On my system, I get the error using both python 3.9.16 and 3.11.1. As you can see from the…
seandavi
  • 2,818
  • 4
  • 25
  • 52
0
votes
0 answers

gcs : chunk_size has no effect in download_as_bytes

I lost in google docs. Using python to download files from gcs. Below is my code source_bucket = storage_client.bucket('mybucket') blob = source_bucket.get_blob('myFileOf1Gb') print('chunk size', blob.chunk_size) # prints None …
Djai
  • 188
  • 10
0
votes
0 answers

Trying to convert to UTF-8 Without BOM in Python for Cloud Function

I have a requirement to read a file from a folder SubFoldinabctentacles in bucket abctentacles (GCS), convert it to UTF-8 without BOM and move it to outbound folder in the same bucket i.e. abctentacles. I have created a cloud function to run the…
vmadhav531
  • 1
  • 2
0
votes
1 answer

How to copy objects across gcp buckets preserving metadata. gsutil cp drops custom metadata keys wilth null values

I need to copy data across Google Cloud Platform - Cloud Storage(GCS) buckets (source is a GCS bucket and destination is a GCS bucket) Since I perform copy along some more operations in small batches I use the gsutil cp command from bash shell…
Yogesh Devi
  • 617
  • 11
  • 30
0
votes
0 answers

How to read a csv/txt file from gcs bucket using Spring Batch?

I want to read a csv/txt file from GCP Storage Bucket using Spring Batch. What are the dependency needed? Please guide me how to achieve this.
Soumik Das
  • 156
  • 11
0
votes
0 answers

To read orc file from GCS bucket

To read orc file from a GCS bucket i'm using below code snippet, where i'm creating hadoop configuration and setting required file system attributes to use gcs bucket val hadoopConf = new Configuration() hadoopConf.set("fs.gs.impl",…
Nitish N Banakar
  • 129
  • 9
0
votes
1 answer

Bash Operator to access GCS bucket file in GCP Astronomer

I need to run some bash operation on a file that has been present in GCS Bucket. bash_operator = BashOperator( task_id='mani_bash', bash_command="""if [ `awk -F: '/^[^HDR][^TRL]/ { print }' gs:////filename.txt | awk -F…
Mani Shankar.S
  • 39
  • 6
1 2 3
…
15 16