Stack Overflow
Stack Overflow

Questions tagged [google-cloud-data-transfer]

Use this tag for Google Cloud Platform Data Transfer products (https://cloud.google.com/products/data-transfer/) which help customers move unstructured data between Google Cloud and other clouds or storage systems in private data centers. This includes products like Storage Transfer Service, Transfer Service for on premises data, Transfer Appliance and the use of the gsutil cp command.

71 questions
0
votes
1 answer

Google Cloud Storage Transfer - Python method for getting status of TransferOperation

There does not appear to be any method in the Python client API for Google's storage transfer service that checks the status of an ongoing transfer operation. There is get_transfer_job, which shows the status of a transfer job itself (and gives the…
Sam Korn
  • 1
  • 1
0
votes
1 answer

Exception in thread "main" com.google.cloud.bigquery.dms.common.exceptions.AgentException: Unable to start tbuild command

I am trying to migrate a table from teradata to BQ using GCP's Data Transfer functionality. I have following the steps suggested on https://cloud.google.com/bigquery-transfer/docs/teradata-migration. A detailed description of the steps is…
Samantha Falcao
  • 1
  • 1
0
votes
1 answer

Update BigQuery scheduled query with notificationPubsubTopic fails

I am using the DataServiceTransferClient API/SDK for Node to create scheduled queries in BigQuery with a notificationPubsubTopic. Creating them works fine, no issues. Updating them results in an error: INVALID_ARGUMENT: notificationPubsubTopic…
callmetwan
  • 1,205
  • 2
  • 14
  • 31
0
votes
1 answer

Defining Month Time Partition using Teradata Data Transfer Service Custom Schema

I am looking for a way to define a Teradata Data Transfer custom schema that implements a month based date partition. The documentation only provides a method to do this at a timestamp or date…
Daniel Zagales
  • 2,948
  • 2
  • 5
  • 18
0
votes
1 answer

Bigquery data transfers from S3 and efficiency of filtering data by last modification datetime

I'm planning a pipeline from an S3 bucket to BigQuery. I have set up daily bigquery data transfers from S3 as described here. So, I have a data transfer configuration that looks daily for files with prefix s3://mybucket/my/path/ and adds them to a…
geo909
  • 394
  • 1
  • 6
  • 18
0
votes
1 answer

Is there any file size limit to transfer data from Storage Transfer service in GCP(Amazon S3 to google cloud)?

I am transferring data from Amazon s3 to google cloud using storage transfer service. Is there any file size limit for the transfer ?
Anonymous_student
  • 5
  • 4
0
votes
0 answers

I am trying google data transfer from azure datalake to GCS and was able to set it up.Could you please let me know why the transfer is very slow?

Google data transfer details issue I am trying google data transfer from azure datalake to GCS and was able to set it up.Could you please let me know why the transfer is very slow?
prasannakumar
  • 1
0
votes
1 answer

Why this sudden error "No URLS matched" when transferring data from a Google cloud bucket?

My data transfer from a google cloud bucket to AWS bucket was in progress but after a few hours I go this error - " Command exception : NO URLs MATCHED" please see the screenshot here.. Any idea why this error and how I can resolve it?
Stanford
  • 31
  • 5
0
votes
1 answer

How to move tables from one dataset to another dataset in Big Query

In the Big Query Data Transfer Service, I have linked Dataset 1 to the Search Ads 360 data transfer. Now I can see 20+ tables in Dataset1. Is it possible to move these 20+ tables to Dataset2 which will be dedicated to Search Ads data.
Awanish Dubey
  • 11
  • 2
0
votes
2 answers

Amazon S3 parquet file - Transferring to GCP / BQ

Good morning everyone. I have a GCS Bucket, which has files that have been transferred from our Amazon S3 bucket. These files are in .gz.parquet format. I am trying to set up a transfer from the GSC bucket to BigQuery with the transfer feature,…
Victoria
  • 1
  • 1
0
votes
1 answer

BigQuery DataTransfer : Invalid constructor input for StartManualTransferRunsRequest

I am trying to trigger a Scheduled Query through a Cloud Function in GCP. I tried to follow several tutorials and examples and went through the documentation too but with the new version of the API google-cloud-bigquery-datatransfer==3.1.1 I can't…
Yohan Obadia
  • 2,552
  • 2
  • 24
  • 31
0
votes
2 answers

Error in GCP BigQuery Data Transfer Service: No suitable driver found for jdbc:redshift://

we followed Google Cloud Platform link https://cloud.google.com/bigquery-transfer/docs/redshift-migration#console to setup Data Transfer from AWS RedShift to BigQuery dataset. But we are getting following error: 18:51:01 Unable to proceed: Could…
Zamir Ansari
  • 11
  • 2
0
votes
0 answers

BigQuery Data Transfer API for Scheduled Query

I am using the Node JS library for BigQuery Data Transfer API to create a scheduled query. I get this error message BigQuery Data Transfer API has not been used in project xxxxxxxxx before or it is disabled. Enable it by visiting.... I visited the…
snowykz
  • 258
  • 1
  • 12
0
votes
1 answer

Bigquery Data Transfer from S3 intermittent success

When using bigquery data transfer to move data into BigQuery from S3, I get intermittent success (I've actually only seen it work correctly one time). The success: 6:00:48 PM Summary: succeeded 1 jobs, failed 0 jobs. 6:00:14 PM Job bqts_5f***…
Gunner
  • 347
  • 3
  • 10
0
votes
3 answers

GCP storage Bucket transfer

I am looking to transfer data from one bucket to another bucket in 2 different google cloud account. Is it possible to transfer data from one GCP Account to another GCP Account? If yes, what are the steps for that ? Also let me know, What are the…
Rohit Saxena
  • 1
  • 1
1 2 3
4
5