Stack Overflow
Stack Overflow

Questions tagged [pybigquery]

23 questions
2
votes
1 answer

Why Temporary GCS bucket is needed to write a dataframe to BigQuery: pyspark

Recently I face an issue while writing the dataframe data into BigQuery using pyspark. Here it was: pyspark.sql.utils.IllegalArgumentException: u'Temporary or persistent GCS bucket must be informed After research the issue I found that Temporary…
Sekhar
  • 627
  • 4
  • 14
  • 34
2
votes
1 answer

Where clause with 80% true Condition in BigQuery?

I have to put a query in BigQuery where we have 100 conditions( in where clause). I want, if any of 80+ conditions are true then it should fetch the data. Is that possible in Big Query?
sanjay
  • 514
  • 2
  • 5
  • 14
1
vote
1 answer

Referenced variable 'ro_sub_ros.$is_not_null' has levels of 1, while the corresponding field path to Parquet column has 0 repeated fields

BigQuery Python: google.api_core.exceptions.BadRequest: 400 Error while reading data, error message: Schema mismatch: referenced variable 'ro_sub_ros.$is_not_null' has array levels of 1, while the corresponding field path to Parquet column has 0…
Shivam Sahil
  • 4,055
  • 3
  • 31
  • 62
1
vote
1 answer

How to know the number of queries per user in Bigquery?

I want to know: how to get the number of queries a user makes per month in BigQuery? how many queries a specific user makes, how many per year. I found the command bq ls -j with that bring the number of jobs and with the comand bq show get the…
Eduardo Rosado
  • 143
  • 1
  • 8
1
vote
1 answer

SQLAlchemy Join using PyBigquery to filter results

Using a SQLAlchemy class, I'm trying to generate a query that resembles SELECT DISTINCT(non_unique_key) FROM `tablename`, UNNEST(tasks_dns) AS dns WHERE create_date_utc = TIMESTAMP("2020-12-31T23:59:59") AND dns LIKE "%whatever%" Being an…
geudrik
  • 666
  • 8
  • 22
0
votes
1 answer

Bigquery library for Python error: Unexpected end of stream: RLE Decoding retrieved less values: 0 then num values: 113

Im having a big time trying to solve this exception on Bigquery library for Python, using the method "load_table_from_dataframe": google.api_core.exceptions.BadRequest: 400 Error while reading data, error message: Failed to import values for column…
Alejandro Dinamarca
  • 3
  • 2
0
votes
1 answer

Why "module 'google.cloud.bigquery' has no attribute 'format_options'"?

I use bigquery fine in my code, but when I try to run this: parquet_options = bigquery.format_options.ParquetOptions() there is an exception: AttributeError("module 'google.cloud.bigquery' has no…
realPro
  • 1,713
  • 3
  • 22
  • 34
0
votes
1 answer

Exporting big query table to GCS to transfer data to a new table

I need to move 20 million records (approx. 10GB) from an unpartitioned big query table to a new partitioned table. The approach is to export the original table to a GCS bucket in JSON format, using wildcard uris. I get 304 json files (approx 21GB)…
Sergio Mujica
  • 141
  • 1
  • 5
0
votes
0 answers

Insert Excel sheet data to an existing Big Query table

We have an existing table on bigquery that gets updated via a scheduler that checks ftp server and upload the new added data into it. The issue is that few days were dropped from the FTP and now I need to upload the data manually into the…
Gojoe
  • 91
  • 1
  • 6
0
votes
0 answers

Maximum list length of the BigQuery ArrayQueryParameter values parameter

I am trying to query BigQuery through the Python API Client Library using query parameters, specifically an ArrayQueryParameter. from google.cloud import bigquery client = bigquery.Client() query = """ SELECT * FROM …
Begona Echavarren
  • 1
  • 1
0
votes
1 answer

Time stamp conversion while inserting data to bigquery

i'm trying to run a insert query at bigquery to insert row, but failing due to timestamp conversion issue, can any one help out here Bigquery table Schema: fieldname type status STRING NULLABLE created_on TIMESTAMP NULLABLE updated_on …
Surya R
  • 445
  • 2
  • 10
0
votes
0 answers

getting BigQuery query status through python

I'm using Google's BigQuery API through Google Colab to build a dashboard holding a few queries. Because some could be "heavy", it can take a minute or two to load the data. I'm looking for a way to show the user a progress bar of some sort, either…
omri
  • 1
  • 3
0
votes
1 answer

Superset BigQuery Issue1011 (Unexpected param: (2022)) when charts executes query containing IN operator

I have been using Superset for the past 2 months and have not experienced this issue before. I have created a number of charts and dashboards that have been functioning as expected for the past 2 months. All my charts pull from BigQuery The base…
Lambo
  • 1,094
  • 11
  • 18
0
votes
2 answers

Google BigQuery - Merge two Queries into one (sql)

In Google BigQuery using SQL, I have 2 separate queries that I want to merge into one. They are both extracting the same fields from the same tables but for different for time, items and location. I don't want any cross over of data from one query…
Petey2005
  • 13
  • 3
0
votes
0 answers

how to stop bigquery randomly deleting columns if credentials are expired

I have a project which uses BigQuery, however each month I have to rotate the credentials. Sometimes I don't get to that in time. BigQuery will then chew up the database, randomly deleting columns. I have to recreate the tables, adding back 30% of…
dcsan
  • 11,333
  • 15
  • 77
  • 118
1
2