Stack Overflow
Stack Overflow

Questions tagged [google-cloud-data-transfer]

Use this tag for Google Cloud Platform Data Transfer products (https://cloud.google.com/products/data-transfer/) which help customers move unstructured data between Google Cloud and other clouds or storage systems in private data centers. This includes products like Storage Transfer Service, Transfer Service for on premises data, Transfer Appliance and the use of the gsutil cp command.

71 questions
0
votes
1 answer

Big Query Data Transfer - How to avoid file limits?

I have a GCS bucket that gets ~10k to 20k new files daily. I want to set up a BQ data transfer to load the new files into my table each day. Given the large amount of files it runs up against the quotas and gives me this error: Error status:…
A Clockwork Orange
  • 23,913
  • 7
  • 25
  • 28
0
votes
2 answers

Max file count using big query data transfer job

I have about 54 000 files in my GCP bucket. When I try to schedule a big query data transfer job to move files from GCP bucket to big query, I am getting the following error: Error code 9 : Transfer Run limits exceeded. Max size: 15.00 TB. Max file…
codechef
  • 65
  • 1
  • 7
0
votes
0 answers

BigQuery Data Transfer error for scheduled queries

Im keep getting this error when I try to go to the scheduled queries dashboard. I have a scheduled query that is importing aggregated data from another project trough a service account. The import seems like it's working for a while but then shortly…
ORCos
  • 89
  • 1
  • 10
0
votes
0 answers

BigQuery Data Transfer Service Error for Cloud SQL

I am trying to use the BigQuery Data Transfer Service by Google Cloud Platform to bring in my data from Cloud SQL (mysql instance) to BigQuery. I have enrolled the data source on the marketplace portal. Also I am using a Public IP for my mysql…
nsk
  • 255
  • 3
  • 12
0
votes
1 answer

Using Transfer for on-premises option to transfer files

[google-cloud-storage]I am trying to copy files from Linux directory to GCP bucket using "Transfer for on-premises" option. I’ve installed docker script on Linux and GCP bucket is created. I now need to run Docker Run command to copy files. My…
numan
  • 1
0
votes
2 answers

Unable to copy 350 GB data from one bucket to another in google cloud

I'm trying to move data from one bucket to another bucket within same project on Google Cloud. It has plain images and documents, like .png, .jpg & .pdf files. I tried several times, but the results remain same. Below is the command i'm…
Mani
  • 29
  • 10
0
votes
1 answer

Google Cloud's Transfer Service for on-premises data

I need to copy data files from a Linux server to Google Cloud Storage buckets and schedule this process to run hourly. I think the "Transfer for on-premises" is the most suitable option for this process. However, I am struggling to find detailed…
numan
  • 1
0
votes
1 answer

Storage Transfer Service To backup Google Firebase Storage

I have a mobile application that stores data in Google firebase database and firebase Storage service. To prevent data loss, I would like to backup the data stored on both services. Google recently launched Autobackups for database but there is no…
user3519594
  • 387
  • 1
  • 11
0
votes
2 answers

Credentials error when using google transfer service to transfer file from AWS to GCP

We have developed an automated pipeline that mainly does tasks on AWS and then some downstream work on Google Cloud. The tasks are deployed on AWS via AWS StepFunctions/Lambda and we need to pass processed files from AWS to Google Cloud Storage (via…
claudiadast
  • 591
  • 3
  • 11
  • 33
0
votes
1 answer

Google storage transfer service with Twilio

I am working with google cloud storage transfer service to transfer data from twilio recording url to google cloud bucket. While implementing above thing I came to know that to transfer a file from url you must have a md5 hash of that object. Now…
swapnil
  • 65
  • 7
-1
votes
0 answers

Scalable software architecture for CCTV real time feed with custom AI model processing in Local Server

I have a project where I need to read video feeds from multiple IP cameras(cctv) simultaneously. These IP cameras are an integral part of our solution, and we aim to capture these feeds and store or view them on a local/cloud server and process them…
Sachin Gaikwad
  • 9
  • 2
1 2 3 4
5