Perform custom SQL query with Google Cloud Data Fusion

Question

I have data pipelines that consist of multiple SQL queries being run against BigQuery tables, I would like to build these in Google Cloud Fusion, but I don't see an option to transform/select with custom SQL.

is this available, or am I misinterpreting the use cases for this tool?

can you please share more details about how you connect from Google Cloud Fusion to BigQuery — Tamir Klein, Apr 18 '19 at 11:13

score 4 · Answer 1 · answered Apr 18 '19 at 14:16

4

A new Action plugin is being added that would allow you to specify a SQL to run in BQ. Expect the connectors to be available in Hub by mid May.

Nitin

answered Apr 18 '19 at 14:16

Nitin Motgi

229
1
3

Can you please share the official page that assures SQL support? – Jaison Apr 18 '19 at 18:01
I can't find this, is it available? – Bjoern Aug 08 '19 at 12:18
1

@BenP this has been merged and would be available next month – Nitin Motgi Oct 13 '19 at 15:26
Thanks @NitinMotgi - what is the name of the action? Is it a part of the GCS addon, or stand alone? – Ben P Oct 16 '19 at 12:38
Anyone know if this has been released yet? – crazy8 Jan 01 '20 at 14:34
1

@crazy8 There is a now a BigQuery action which fulfills this requirement. – Ben P Jan 02 '20 at 08:58

score 2 · Accepted Answer · answered Jan 02 '20 at 09:36

There is now a native BigQuery Execute action that allows SQL queries to run as part of a Data Fusion Pipeline.

This job is an action, see below from the official documentation:

Action plugins define custom actions that are scheduled to take place during a workflow but don't directly manipulate data in the workflow. For example, using the Database custom action, you can run an arbitrary database command at the end of your pipeline. Alternatively, you can trigger an action to move files within Cloud Storage.

Perform custom SQL query with Google Cloud Data Fusion

2 Answers2