0

I have been trying multiple ways to move the compressed TSV to Big query. I was able to get the command working but didn't see any table being loaded. Please help me figure out to write the command that works.

bq ‘--project_id’ --nosync load --source_format CSV --field_delimiter ‘\t’ --autodetect --skip_leading_rows ‘0’ --quote=‘’ --encoding UTF-8 :table.destinationtable ‘gs://bucketname/filename.tsv.gz’

Successfully started load 162822:bqjob_r2d00a5817904935f_0000015c79e61b7c_1

Ramya
  • 21
  • 4
  • 1
    What do you mean that you "didn't see any table being loaded"? Loading isn't instantaneous, so you need to check the status of the job to see when it finishes. – Elliott Brossard Jun 13 '17 at 14:03
  • @ElliottBrossard I was waiting for a while to check the destination table and it was not having any values. – Ramya Jun 13 '17 at 14:36
  • You need to [check the status of the load job](https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get). You can use the "Try this API" at the bottom of the page. – Elliott Brossard Jun 13 '17 at 14:58
  • I tried the API, got the error message below. {"reason": "invalid", "location": "gs:///.tsv.gz", "message": "Could not parse 'ibcx' as int for field int64_field_5 (position 5) starting at location 3743224 " } – Ramya Jun 13 '17 at 17:15
  • Okay, so now you have a pointer to the problem that you need to fix before trying again. Alternatively, you can set the [`configuration.load.maxBadRecords` property](https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.maxBadRecords). – Elliott Brossard Jun 13 '17 at 17:23
  • Does this error message mean the tsv file has data formatting issues or an extra tab in some row? How can that be handled while importing? – Ramya Jun 13 '17 at 17:58
  • Check the other configuration options for jagged rows, etc. It sounds like you just have a bad value in a row in this case. – Elliott Brossard Jun 13 '17 at 18:51
  • I tried both, --max_bad_records and --allow_jagged_rows. The load operation ran for a while and I got error "status": { "state": "DONE", "errorResult": { "reason": "internalError", "message": "An internal error has occurred" }, "errors": [ { "reason": "internalError", "message": "An internal error has occurred" } ] }, "statistics": { "creationTime": "1497382116337", "startTime": "1497382116976", "endTime": "1497382817991" }, – Ramya Jun 13 '17 at 19:43
  • This looks like a bug, so consider [posting a bug on the issue tracker](https://issuetracker.google.com/issues/new?component=187149&template=0) with a job ID and someone can look into it. – Elliott Brossard Jun 13 '17 at 20:29

0 Answers0