Stack Overflow
Stack Overflow

Questions tagged [google-cloud-data-fusion]

Google Cloud Data Fusion is a fully managed, cloud-native data integration service that helps users efficiently build and manage ETL/ELT data pipelines. Data Fusion has a visual point-and-click interface, transformation blueprints, and connectors to make ETL pipeline development fast and easy. Cloud Data Fusion is based on the open-source CDAP project.

Google Cloud Data Fusion is a fully managed, cloud-native data integration service that helps users efficiently build and manage ETL/ELT data pipelines. Data Fusion has a visual point-and-click interface, transformation blueprints, and connectors to make ETL pipeline development fast and easy. Cloud Data Fusion is based on the open-source CDAP project.

This tag can be added to any questions related to using/troubleshooting Google Cloud Data Fusion.

Useful links:

445 questions
2
votes
1 answer

SQL Server Source in Google Data Fusion Doesn't Work (SSL handshake issue)

I created a simple pipeline in Data Fusion, which reads from a single MS SQL Server table and writes to BigQuery. It fails when connecting to the SQL Server with a socket handshake error. I've seen this issue when creating my own dataproc clusters,…
Bjoern
  • 433
  • 3
  • 16
2
votes
2 answers

How to pass schema file as Macros to BigQuery sink in data fusion

I am creating a data fusion pipeline to load csv data from GCS to BigQuery for my use case i need to create a property macros and provide the value during runtime. Need to understand how we can pass the schema file as Macros to BigQuery sink. If i…
Mustaquim
  • 103
  • 5
2
votes
1 answer

Can anyone help me with this error code in Data Fusion

I'm having a go at creating my first data fusion pipeline. The data is going from Google Cloud Storage csv file to Big Query. I have created the pipeline and carried out a preview run which was successful but after deployment trying to run resulted…
mez63
  • 166
  • 4
  • 20
2
votes
1 answer

java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.(Ljava/io/InputStream;Z)V in preview

I'm experimenting with Google Cloud Data Fusion. I'm joining 2 BigQuery tables using the joiner + writing back to BigQuery. In preview I get this error : java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.(Ljava/io/InputStream;Z)V…
Koen Verschaeren
  • 73
  • 5
2
votes
2 answers

Google Cloud Data Fusion 1. Does not connect to oracle 2. When the pipeline is running I get 'default' network port error

I installed oracle-jdbc thin driver to connect with On prem oracle DB but when I test the connection I get network adapter error I tried the changing the host but still same When running the pipeline from GCS-BQ I getting network port error. Can…
Sailesh
  • 21
  • 2
2
votes
1 answer

BigQuery View is not working if I used BigQuery Plugin

I've been used bigquery plugin under the source category. When I used bigquery View, Pipeline through an error of not allowed View. Also If I used the permanent table in which repeatable columns have existed, then it also through an error of…
Ashish Balhara
  • 63
  • 1
  • 5
2
votes
1 answer

Outputted CSV file is splitting up with Google Cloud Storage Sink Plugin

I've been used Google Cloud Storage Plugin under the Sink category in a pipeline, to get the output in CSV format. After the execution of the pipeline, the resulted output is coming in several files after splitting up. Is it the right behaviour of…
Ashish Balhara
  • 63
  • 1
  • 5
2
votes
1 answer

Failed to deploy Cloud Data Fusion "Transfer Data From MySQL to Google BigQuery" template pipeline

I've just tried Cloud Data Fusion, Transfer Data From MySQL to Google BigQuery template pipeline. After the configuration of the setting, I got the error below while deploying the pipeline. Any advice? Plugin named 'BigQueryTable' of type…
Yuki Nagae
  • 21
  • 1
2
votes
2 answers

How to use Custom Transform in Wrangler?

I'm trying to use custom transform using a column at Wrangler Plugin. Is there any documentation where I can find the list of functions to apply the custom transform? Also for a specific case, I wanna replace the value of a column based on IF-ELSE…
Ashish Balhara
  • 63
  • 1
  • 5
2
votes
1 answer

How to edit directives previously entered?

I am new to CDAP / Data Fusion. I have created a preparation / wrangler which has included a set of directives. I find that I now wish to edit these previously entered directives but can't seem to find out how to perform this task. For example, in…
Kolban
  • 13,794
  • 3
  • 38
  • 60
2
votes
2 answers

How to schedule Google Data Fusion pipeline?

I have deployed a simple Data Fusion pipeline that reads from GCS and writes to BigQuery table. I am looking for way to schedule the pipeline but could not find relevant documents. Can anyone point me to documentation/pages that briefs about…
Jaison
  • 715
  • 1
  • 10
  • 33
1
vote
1 answer

Public Cloud SQL, private Data Fusion instance

I am new to GCP, so I got some questions regarding Cloud SQL. From what I read I need to install a proxy once I want a private Data Fusion instance to connect to a private Cloud SQL instance. But do I also need one, if I want to connect a private…
Sholly
  • 23
  • 5
1
vote
1 answer

Error when pulling data from SAP data source using table connector

EDIT: I am using Cloud Fusion (CDAP) on GCP. When validating a data pipeline that pulls data from SAP using the table connector, I get this error: Invalid schema: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 2 path $ I…
Matteo Fiorillo
  • 11
  • 1
1
vote
1 answer

mysql-connector-python 8.0.33 has requirement protobuf<=3.20.3,>=3.11.0, but you have protobuf 4.23.1

Using GCP Cloud Composer,trying to add PyPI package google-cloud-data-fusion Version used: composer-2.2.0-airflow-2.5.1 Issue: trying to add PYPI "google-cloud-data-fusion" and it's throwing the error. UPDATE operation on this environment failed…
A_Sk
  • 4,532
  • 3
  • 27
  • 51
1
vote
1 answer

Data Fusion for xml-to-json transformation: "+ExitOnOutOfMemoryError" and "exited with a non-zero exit code 3. Error file: prelaunch.err"

When transforming an xml file to json, the Data Fusion pipeline, configured in Autoscaling mode up to 84 cores, stops indicating an error. Can anybody help me to make it work? The 100-pages Raw log file seems indicating that possible errors…
Mauro Di Pasquale
  • 13
  • 2
1 2 3
…
29 30