Stack Overflow
Stack Overflow

Questions tagged [google-cloud-data-transfer]

Use this tag for Google Cloud Platform Data Transfer products (https://cloud.google.com/products/data-transfer/) which help customers move unstructured data between Google Cloud and other clouds or storage systems in private data centers. This includes products like Storage Transfer Service, Transfer Service for on premises data, Transfer Appliance and the use of the gsutil cp command.

71 questions
1
vote
1 answer

Duplicated data in Google BigQuery Google Ads Data Transfer Tables

I have configured a Google Ads DataTransfer stream from Google Ads to my GoogleBigQuery project. It runs, data flows, everything is fine. But when I decided to build a query that return an amount of money spend in the context of distinct combination…
Roman
  • 87
  • 5
1
vote
1 answer

Data Transfer from BQ To CLoud SQL

what is best way to transfer all records from BigQuery table to Cloud SQL table on daily basis (every day expected approximate count of records more than 255801312 [255 million]). I know we can create dataflow pipelines from BQ to CloudSQL, but this…
Srinivas B
  • 13
  • 4
1
vote
2 answers

GCP Cloud Data transfer - any notifications?

I want to build a small solution that will transfer files from Azure to GCP. The solution itself works fine, but I don't see any solution anywhere that could inform me that the transfer started, ended or there was an error (e.g. on Slack). I can't…
Kivas
  • 37
  • 9
1
vote
2 answers

Setting up a scheduled query with a service account error PermissionDenied: 403 The caller does not have permission in data_transfer_service

I am trying to schedule query using big query data transfer api and giving required permission bigquery.admin and enabled the big query transfer api. Permission…
iamabhaykmr
  • 1,803
  • 3
  • 24
  • 49
1
vote
1 answer

BigQuery Data transfer jobs failed

BigQuery data transfer job failed with following error message. "Error code 7 : Invalid account; The issue may be resolved by verifying that the user has sufficient permissions and updating the credentials for this transfer config; error:…
SKumar
  • 43
  • 6
1
vote
0 answers

Using GCP Data Transfer Service to transfer files from S3 previously stored in glacier

Was using the GCP data transfer service to transfer a set of files in an S3 bucket over to a GCP storage bucket. The files were in glacier so I first restored them and then tried copying them. The transfer job runs without any errors, but ignores…
Joe Slagel
  • 9
  • 1
1
vote
2 answers

BigQuery - data transfer success but destination table stays empty

Final goal: transfer csv file contents from google cloud storage bucket to a table in BigQuery dataset. problem: data transfer seems to be working, not showing any errors/warnings but the data fails to reach the destination table. dataset/table…
sivan
  • 81
  • 1
  • 1
  • 7
1
vote
1 answer

Retain public access to files when using Google Cloud Storage Transfer Service

I use Google Cloud Storage Transfer Service to move files between Google cloud buckets. I've actually executed one transfer job. In the source bucket there were files with enabled public access. In the destination bucket the same files no longer had…
Denny
  • 116
  • 1
  • 6
1
vote
1 answer

Why does querying a report from google play console by the google cloud BigQuery API give incomplete results

I'm trying to get data from one of the reports available in the google play console. Specifically the user_acquisition report. I set up the data transfer service within the google cloud platform in order to use the BigQuery API. When querying that…
M.K
  • 11
  • 2
1
vote
2 answers

Google Cloud Storage Transfer Failed Files Not Listed

We recently did an "on-premises transfer" from an old server being decommissioned into a Google Cloud Storage bucket connected via docker. The total size of the transfer was 365GB. The console shows that the transfer was 100% complete with 365GB…
VirtualMitra
  • 93
  • 7
1
vote
2 answers

Migrating data from Google Cloud Platform to AWS S3

I tried to migrate data from Google Cloud Platform, I tried this method by Mike Exporting data from Google Cloud Storage to Amazon S3, but got the complaint, anyone knows what should I do next?
Shuai Shao
  • 21
  • 4
1
vote
3 answers

Amazon equivalent of Google Storage Transfer Service

I have a bucket in GCP that has millions of 3kb files, and I want to copy them over to an S3 bucket. I know google has a super fast transfer service, however I am not able to use that solution to push data back to S3 with it. Due to the amount of…
dwardu
  • 573
  • 6
  • 22
1
vote
2 answers

Google Storage Transfer Service cost-efficiency

I need to backup my aws s3 bucket with lots of files (800k files, 400GB) to google cloud storage. Google Storage Transfer Service seems like an easy solution but i have some doubts about cost-efficiency. First time it will download a whole s3…
Sergey Babochkin
  • 173
  • 3
  • 7
1
vote
1 answer

Google transfer service error notification

I've been looking everywhere and can't seem to find the answer. I set up a file transfer service between a S3 bucket and a google storage bucket. I know I can see the error messages by clicking on the file transfer, but I wan't to access the log, so…
Bas
  • 203
  • 1
  • 3
  • 10
1
vote
1 answer

How can I use the Storage Transfer Service to copy data across buckets in different projects?

I'm attempting to shuttle data between buckets in different projects using Google Cloud's Storage Transfer Service based on an event that's being emitted. This event points to a subdirectory of a bucket belonging to project A that I want to copy to…
Chester Husk
  • 1,408
  • 12
  • 14
1
2
3 4 5