0

I want to ingest data to BigQuery from my java application. Is there any performance issue if we are using BigQuery API directly ? Application is running in AWS.

Faisal P P
  • 111
  • 1
  • 9

1 Answers1

1

If performance is a concern, you can use BigQuery streaming inserts with a Java API [1,2]. [3] outlines some of the quota limitations.

[1] https://cloud.google.com/bigquery/streaming-data-into-bigquery

[2] https://cloud.google.com/blog/big-data/2017/06/life-of-a-bigquery-streaming-insert

[3] https://cloud.google.com/bigquery/quotas#streaminginserts

Chris Hill
  • 121
  • 4
  • My actual use case is, we are running an email application whenever we get a new message we are processing it for some data and store to BQ. if we call insertAll api every time we get a message, that means we need to call insertAll everytime we get a new message, is it ok to call insertAll like that ? – Faisal P P Dec 18 '17 at 06:03