-1

I'm using python client library to insert data to big query table. The code is as follows.

client = bigquery.Client(project_id)
errors = client.insert_rows_json(table=tablename,json_rows=data_to_insert)
assert errors == []

There are no errors, but the data is also not getting inserted.

Sample JSON rows:

[{'a':'b','c':'d'},{'a':'f','q':'r'},.....}]

What's the problem? No exception also

bigbounty
  • 16,526
  • 5
  • 37
  • 65
  • The doc says "Row data to be inserted. Keys must match the table schema fields and values must be JSON-compatible representations." Does it match? Can you show the schema please? – knh190 May 27 '19 at 10:24
  • This [issue](https://github.com/googleapis/google-cloud-python/issues/5171) also mentions the freshly create table will not get updated immediately. – knh190 May 27 '19 at 10:27
  • Also, I'm not quite sure if your code is complete, have you seen [this](https://github.com/googleapis/google-cloud-python/blob/cf04f6d15615b09c1de9e584bb3462653f0770f1/bigquery/docs/snippets.py#L1180) example? If there's more to your code please update your question. The table's schema would also be useful as knh190 mentions. – ch_mike Jun 07 '19 at 15:13

3 Answers3

2

client.insert_rows_json method using StreamingInsert . Inserting data to BigQuery using StreamingInsert will be cause of latency on table preview on BigQuery console.
The data is not appeared immediately. So, You need to query them to confirm the data inserted.

takryo
  • 429
  • 4
  • 8
1

It can be 2 possible situations:

  1. your data does not match the schema
  2. your table is freshly created, and the update is just not yet available

References:

knh190
  • 2,744
  • 1
  • 16
  • 30
0

got the answer to my question. The problem was I was inserting one more column data for which data was not there. I found a hack in order to find out if the data is not inserting to bigquery table.

  1. Change the data to newline delimited json with the keys as the column names and values as values you want for that particular column.
  2. bq --location=US load --source_format=NEWLINE_DELIMITED_JSON dataset.tablename newline_delimited_json_file.json. Run this command in you terminal and see if throws any errors. If it throws an error it's likely that something is wrong with your data/table schema.
  3. Change the data/table schema as per the error and retry inserting the same via python.

It's better if the python API throws an error/exception like on the terminal, it would be helpful.

bigbounty
  • 16,526
  • 5
  • 37
  • 65
  • Good to know you were able to figure this out. Based on your latest comment, I would recommend you to open a feature request against the python library so that it's development team could be aware of the enhancement you suggest. You can use this [link](https://github.com/googleapis/google-cloud-python/issues) to the GCP python libraries issues on Github. – ch_mike Jun 10 '19 at 15:50