0

I'm using KNIME's Database Connection & Database Writer in combination with Simba's JDBC driver to try uploading a dataset as a BigQuery table.

However, I don't get it working. For example, I don't know where to specify a BiqQuery dataset. Do I have to add it in the connection URL in the Database Connection node? And if so, do I need a new parameter?

Does someone has experience uploading data from KNIME to BigQuery or knows a workaround, for example with GCS?

1 Answers1

0

when you download SIMBA driver you will find within folder pdf file named Simba JDBC Driver for Google BigQuery Install and Configuration Guide. There you will find that connection URL depends on Authentication type you use. You will see that you do not specify dataset in Database Connection node but only ProjectId. Within Database Writer node you will specify dataset and table.

Here you can find more on loading data in BigQuery: https://cloud.google.com/bigquery/docs/loading-data

From my experience I did not find it easy to load data into BigQuery anyhow.

ipazin
  • 102
  • 7
  • thanks for your comment! Was wondering how I am able to specify dataset in the "Database Writer node". Adding dataset information in the table name section is not working... However, you were not able to get it working properly? Did you find a workaround? – Guido Stut Jul 22 '19 at 08:36
  • You specify dataset same as schema. So if dataset is called _test_ and table _numbers_ you will write _test.numbers_ in Database Writer node for Table Name. I got it to work properly but batch size can be only 1 (for driver I got) so writing lot of data is not feasible. A possible workaround could be to use KNIME External Tool node either to transfer your data to Google Cloud Storage using gsutil cp command and from there create a job to transfer data to BigQuery or use bq command-line tool 1 directly to load data. Disclaimer: Haven’t tried it out so feel free to share your experience with us. – ipazin Jul 23 '19 at 08:30