Stack Overflow
Stack Overflow

Questions tagged [google-bigquery-java]

15 questions
1
vote
1 answer

BQ load from csv file stored in storage using java client

Tried to load into existing BQ table from csv file stored in the bucket using google BQ client but ran into this error. Not sure how can I figure out this position in the csv file. I replaced all the NULL strings with "" values and also surrounding…
gcpdev-guy
  • 460
  • 1
  • 10
  • 23
1
vote
0 answers

Big query table on top of bigtable taking too long to read in google dataflow job

I have a dataflow job that reads from bigquery table( created on top of big table). The data flow job is created using custom template in java. I need to process around 500 million records from bigquery. The issue I am facing is even to read 1…
Ankit Gautam
  • 125
  • 1
  • 11
1
vote
0 answers

BigQuery Create Table with Structs using Java API

I am trying to create a table in Big Query using Java API for the below format col1 STRUCT>> . I am able to create like col2 ARRAY> , but unable to wrap another STRUCT outside of…
1stenjoydmoment
  • 229
  • 3
  • 14
0
votes
0 answers

What is the cost of querying BigQuery REST APIs. Specifically: projects.locations.searchLinks

I'm trying to use the searchLinks API provided by Google BigQuery, but they have no information on how much it costs. Looking for any pointers on how to think about the cost for using the API. API…
Pramod Kalipatnapu
  • 41
  • 3
0
votes
1 answer

SQL query to join tables with duplicate rows and getting unique rows

I got the following three tables in BigQuery along with the data below. I'd like to write a SQL query using id and current date columns and get the following row as the output. Expected result set: id existing_stores …
gcpdev-guy
  • 460
  • 1
  • 10
  • 23
0
votes
1 answer

Integration tests with BigQuery's JsonStreamWriter in Java

For local development and integration testing, I am using a BigQuery emulator that exposes ports 9050 (for HTTP) and 9060 (for grpc). For example, this is how I instantiate BigQuery client object in Java (this works and it is connecting to my…
Emanuel Trandafir
  • 1,526
  • 1
  • 5
  • 13
0
votes
0 answers

BigQuery - accessing data with OAuth credentials

How can I interact with Bigquery using OAuth credentials (refresh tokens)? I've managed to do it using classes from the com.google.api.services.bigquery package: import java.io.IOException; import java.util.ArrayList; import java.util.Date; import…
Buffalo
  • 3,861
  • 8
  • 44
  • 69
0
votes
0 answers

Are there Testconatiners for BigQuery emaulator?

I am working with the Java BigQuery API Client and I am not sure how to write some integration tests for it. I saw this old StackOverflow post but I don't know If anything has changed since then. I can see testcontaienrs for some of the GCloud…
Emanuel Trandafir
  • 1,526
  • 1
  • 5
  • 13
0
votes
0 answers

BigQuery - can a project be created using the API?

I'm using com.google.api.services.bigquery.Bigquery and the projects() method only has list(): bigquery.projects().list();. How to create a project using the API?
Buffalo
  • 3,861
  • 8
  • 44
  • 69
0
votes
0 answers

Can I use "WRITE_APPEND" option with insertAll() method of BigQuery class?

I'm using BigQuery java library to send data to BigQuery table. I have list of data to send and I'm using bigQuery.insertAll() method to send all that data at once but I wanted to get rid of duplicates with the same timestamp so I added insertId…
dominosski
  • 1
0
votes
1 answer

Fields of type Record are inserted as null in BigQuery

I have a Scala job where I need to insert nested JSON file to BigQuery. The solution for that is to create a BQ table with field type as Record for the nested fields. I wrote a case class that looks like this: case class AvailabilityRecord( …
Idhem
  • 880
  • 1
  • 9
  • 22
0
votes
0 answers

Is there an API to get the names/number of files created by Bigquery "EXPORT DATA" statement?

I am constructing "Export Data" statement and submitting it as Query to Bigquery using BigQuery API Services. To specify job configuration, I am using "JobConfigurationQuery". So "Export Data" statement query looks like as below: Ref: EXPORT DATA…
ramanKC
  • 197
  • 1
  • 3
  • 16
0
votes
1 answer

How to setTimeout for JsonStreamWrite api

I am having a Java Spring boot application which uses JsonStreamWrite of Google BigQuery Storage Write API to write data to BigQuery. I wanted to timeout the stream write if it takes more than a minute to insert into BigQuery. Here is my sample…
Vijay Manohar
  • 473
  • 1
  • 7
  • 22
0
votes
1 answer

BigQuery Streaming Insertion takes time in minutes for write-heavy application

I have an write-heavy Springboot application integrating with Bigquery for heavy load , facing 10 minutes to insert some of the entries.Here are my configurations Number of Entries Stored: 1 Million/min Number of pods : 100 Insertion Type :…
Vijay Manohar
  • 473
  • 1
  • 7
  • 22
0
votes
0 answers

BigQuery Storage Api row restrictions on clustered table: do they reduce cost?

I'm using the BigQuery Storage Api java client (from scala), reading from a clustered table having 4 clustered fields, e.g. bq mk [...] --table --clustering_fields f1,f2,f3,f4 mytable mytableschema.json and the schema looks like the following [ { …
Giovanni Caporaletti
  • 5,426
  • 2
  • 26
  • 39