Stack Overflow
Stack Overflow

Questions tagged [bq]

bq is a python-based, command-line tool for BigQuery. It allows the creation of datasets, tables, query jobs; and listing and exporting data among other utilities using the command-line.

bq is a python-based, command-line tool for BigQuery. It allows the creation of datasets, tables, query jobs; and listing and exporting data among other utilities using the command-line.

104 questions
0
votes
1 answer

List of Months with Years

How can I simply get a list of months with years without a field? Kind of like how we do current_date() but instead a list of month_years to date in SQL/Big Query
Panda48
  • 33
  • 4
0
votes
1 answer

JSON Errors when trying to create a transfer_config in Google BigQuery CLI

I am trying to create a transfer job on the windows commandline with bq mk --transfer_config --data_source=amazon_s3 --target_dataset=Usage --display_name='s3_transfer_installs_global_in_v0_test' --params='{"data_path_template":"mybucket",…
Manuel Huppertz
  • 310
  • 2
  • 9
0
votes
2 answers

Export Google BigQuery datasets in a single project as list in bq command-line?

I've looked at this resource, but it's not quite what I need. This question is what I want to accomplish, but I want to run it in the BQ terminal. For instance, in the past I've exported table information as a .json in bq command-line as so: bq…
s-mabdurrazak
  • 64
  • 6
0
votes
1 answer

Loading data from BigQuery into Google Sheets using the command line

I am looking for a way to upload data from Big Query into Google Sheets using the bq command line. Could not find any relevant documentation on the topic. Thanks in advance.
denim
  • 427
  • 4
  • 16
0
votes
1 answer

How to dynamically change the table schema while reading csv data in BQ?

I have data in txt format with several files, each file has 3 columns and few files have 4 columns, how do create the table which will only read 1st 3 columns? i have been using BQ UI for creating tables, this is causing error when reading data with…
abhimanyu
  • 123
  • 1
  • 8
0
votes
2 answers

GCP Bigquery Error in Load Operation: Bytes are Missing

I am very new to Google Cloud Platform and I'm trying to create a table in bigquery from ~60,000 csv.gz files stored in a GCP bucket. To do this, I've opened Cloud Shell, and I'm trying the following: $ bq --location=US mk my_data $ bq --location=US…
morepenguins
  • 1,187
  • 10
  • 21
0
votes
1 answer

How to load multiple files (same schema) from LOCAL into a table in BigQuery?

I have multi-files in a folder in my local machine. And every file has the same schema. how can I upload those files into BQ in one line cli? I tried this: bq load --source_format=NEWLINE_DELIMITED_JSON --ignore_unknown_values temp.test_load ./*…
emily.mi
  • 161
  • 12
0
votes
1 answer

google cloud sdk is installed in my EC2 instance and I could access gcloud. But bq is not available even though I see it in the list of components

I am installed google-cloud-sdk in my matillion instance hosted on EC2. I am able to access gcloud command in the ssh instance and also by using a bash component in my matillion. However, I am not able to run bq commands. I see it has been installed…
usersas
  • 27
  • 5
0
votes
2 answers

Google CLI does not create Transfer Service and shows no detailed error

I'm very new to Google Cloud CLI, sorry for the dumb question but it's really annoying. I'm trying to execute this command: bq mk --transfer_config --target_dataset=ds_scrummate --display_name='map_email' --data_source=amazon_s3…
elaspog
  • 1,635
  • 3
  • 21
  • 51
0
votes
1 answer

BigQuery cli bq how to use EXECUTE IMMEDIATE script

I would like to use two SQL scripts to create and update a table in BigQuery using bq query command. Here are the two scripts : 1- Creating the table EXECUTE IMMEDIATE '''CREATE TABLE IF NOT EXISTS dataset.newtable ( id STRING, ''' || …
muslash
  • 39
  • 6
0
votes
1 answer

How can I read google-cloud-storage file's metadata from BigQuery

I have a csv file in my google-cloud storage and I have some custom meta-data on this file. I want to read this metadata from my BigQuery. Is this possible. If so please let me know how this should be done.
George's
  • 1
0
votes
1 answer

Retrieve SQL query string from BigQuery query for jobId from two months ago

I'm trying to get the SQL query string for a custom query executed in BigQuery on a specific day from nearly two months ago. I've executed the following command to retrieve a list of all unique values of jobId from that day: bq ls --jobs --all…
mrag
  • 51
  • 3
  • 10
0
votes
1 answer

Missing hive partition key column while creating hive partition external table using bq command

I am using the bq version 2.0.58 and trying to create hive partitioning BigQuery external table. But it's missing hive partition key column while creating hive partition external table using bq command, the following command to create a…
SST
  • 2,054
  • 5
  • 35
  • 65
0
votes
1 answer

External Bigquery hive partition table created for dataset but not able to view the column with partition key

I have created a terraform script for creating a dataset and External table in Bigquery. Below is the terraform script 1 resource "google_bigquery_dataset" "bigquery" { 2 dataset_id = "${var.biqquery_dataset_id}" 3 …
klee
  • 1,554
  • 2
  • 19
  • 31
0
votes
2 answers

bq extract - BigQuery error in extract operation: An internal error occurred and the request could not be completed

I am trying to export a table from BigQuery to google storage using the following command within the console: bq --location= extract --destination_format CSV --compression GZIP --field_delimiter "|" --print_header=true…
Imad
  • 2,358
  • 5
  • 26
  • 55
1 2 3 4 5
6
7