Stack Overflow
Stack Overflow

Questions tagged [bq]

bq is a python-based, command-line tool for BigQuery. It allows the creation of datasets, tables, query jobs; and listing and exporting data among other utilities using the command-line.

bq is a python-based, command-line tool for BigQuery. It allows the creation of datasets, tables, query jobs; and listing and exporting data among other utilities using the command-line.

104 questions
1
vote
0 answers

How to list user roles/permission on dataset/tables in Big query including inherited roles/permissions?

Is there any way through which we can list all the permissions on the dataset/table without using the bq console , I would like to achieve it using bq command or bq Sql. I know through bq show 'dataset_name' we can list the roles/permissions but it…
Mayur7771
  • 11
  • 1
1
vote
2 answers

BQ query, not exists/joins

in a BigQuery I have a table. VM Package VM1 A VM2 B VM3 A VM4 B VM1 B VM2 C VM3 B VM4 C How can I get results, so all distinct VMs would be listed, but having Package column with value null (or empty, or Yes and No) if…
NewUser
  • 25
  • 1
  • 7
1
vote
0 answers

How does the BigQuery command line tool "bq query" stop printing the sql query string for some jobs?

Let's say the following file exists in the current directory: # example1.sql SELECT 1 AS data When I run bq query --use_legacy_sql=false --format=csv < example1.sql I get the expected output: Waiting on bqjob_r1cc6530a5317399f_0000017f2b966269_1…
Moritz Wolff
  • 436
  • 1
  • 7
  • 16
1
vote
1 answer

Bigquery - Version Control Scheduled Queries

Right now I have scheduled queries via BQ interface. They work but do not scale or migrate very well (across dev and prod gcp projects). So I am trying to do scheduled queries in a way that is reproducible, scalable and migrate-able. My queries are…
Urvah Shabbir
  • 945
  • 2
  • 15
  • 46
1
vote
1 answer

Push JSON file with special chars unicode to bigquery via command line

Im trying to push a json file to BQ via bqcmd which contains special chars such as ü but tis failing. Ive tried differnt encoding etc. nothing workign except removing the chars { "productName": "soft drüum", "Category": "xxxxxxx", …
davey
  • 1,544
  • 4
  • 26
  • 41
1
vote
1 answer

how to achieve python task using local bq operator from airflow

while working on project came across one situation where we want to perform few task using airflow but we are not allowed to use python operator but instructed to use local BigQuery operators. can anyone help me with such code like to setup airflow…
Yug
  • 105
  • 1
  • 9
1
vote
1 answer

Data Transfer from BQ To CLoud SQL

what is best way to transfer all records from BigQuery table to Cloud SQL table on daily basis (every day expected approximate count of records more than 255801312 [255 million]). I know we can create dataflow pipelines from BQ to CloudSQL, but this…
Srinivas B
  • 13
  • 4
1
vote
1 answer

Streaming events to Big Query - Dataflow - Best way to insert **epoch timestamp** (int) to timestamp column

I am trying to write events in streaming into Big Query table using dataflow apache beam (python) having a problem with timestamp format. I have an event (json) with epoch timestamp (int) values which I want to insert into BIG QUERY table with…
anats
  • 11
  • 2
1
vote
2 answers

Code fails to update a table on BQ using DML, but succeeds for insertion and deletion with RPC

I wrote some code that uses service-account to write to BQ on google-cloud. A very strange thing is that only "update" operation using DML fails. (Other insertion, deletion RPC calls succeeds). def create_table(self, table_id, schema): …
Elad Benda
  • 35,076
  • 87
  • 265
  • 471
1
vote
0 answers

splitting values of a row into column in BQ but other column values should repeated

I have a table like below - Project Col-A Col-B Col-C A a,b,c,d e,f g B d,e a,b,c f & output I want as…
Swetosree Sinha
  • 31
  • 3
1
vote
1 answer

BigQuery CLI show tableId only

I want to get only the list of tables in a dataset. bq ls dataset command show the list of table name along with extra columns which are Type, Labels, Time, Partitioning and Clustered Fields. How can I only show the tableId column?
Ali
  • 7,810
  • 12
  • 42
  • 65
1
vote
1 answer

Split string by comma in BigQuery

I would like to split a string in the BQ row into other rows by comma. But I need to split only in case if it is just a comma without space. As of now, I use, for example, SELECT SPLIT('Mon,Tue,Wed', ',') but If I have 'Mon, 23,Tue,Wed, 25' I would…
m.k
  • 83
  • 7
1
vote
2 answers

How can I import CSV data with newline characters in its values into a GCloud Big Query table?

My bq load command and error $ bq load --replace --source_format=CSV --autodetect temp.BQTable ./source.csv Upload complete. Waiting on bqjob_123 ... (0s) Current status: DONE BigQuery error in load operation: Error processing job…
James T.
  • 910
  • 1
  • 11
  • 24
1
vote
1 answer

BIGQUERY Sum Col1 based on unique entries in Col2 and on specific date

I am trying to sum a column1 (invoice_value) in BQ based on a specific date but I want to avoid the duplicates in Column2 (invoice_no). So far I can sum the column1, but the total sum I get includes several duplicates in column2 (invoice_no) SELECT…
PG1
  • 13
  • 2
1
vote
1 answer

Does CREATE OR REPLACE in BigQuery contribute towards a previous Partition Table's quota?

We have a BigQuery query like: create or replace table `{project}`.`{dataset}`.`{table}` partition by date select {...} If we run this query a few times in a day, we get an error: Quota exceeded: Your table exceeded quota for Number of…
Maximilian
  • 7,512
  • 3
  • 50
  • 63
1
2
3 4 5 6 7