Stream data into Google BigQuery using GET Method?

Question

I need a good solution, preferably existing one, such as Google Rest API, for data to stream/insert into BigQuery. I don't want to use POST method to send data - for many design reasons. I am expecting 1000s of writes per seconds. The data will be appended in url's query parameters. It's very similar to what Google analytics does, sending analytics data using Get method.

score 0 · Answer 1 · answered Oct 12 '17 at 12:24

0

Here have a gift:

http://dmitriilin.com/exporting-data-google-analytics-google-bigquery/ https://github.com/lnklnklnk/ga-bq

If you have any extra question ask me, i worked with this solution.

answered Oct 12 '17 at 12:24

Marco

1,112
1
14
34

Thanks Marco, but I ended up using Google Cloud functions, which solved the problem. – ituring Oct 31 '17 at 10:01

score 0 · Answer 2 · answered Oct 13 '17 at 20:59

0

I might also encourage you to use something like a Dataflow streaming pipeline to handle those inserts, rather than calling the API directly.

answered Oct 13 '17 at 20:59

Yuri Grinshteyn

727
3
13

score 0 · Accepted Answer · answered Oct 31 '17 at 10:05

Google cloud functions (GCF) solved the problem. GCF is still in beta but well most google products are in beta anyway. GCF is serverless and one can write http triggers in few line of codes. For further information visit -> https://console.cloud.google.com/functions

Stream data into Google BigQuery using GET Method?

3 Answers3