Getting error while using google data transfer to transfer multiple Json file from GCSto Bigquery

Asked Dec 08 '21 at 10:32

Active Dec 08 '21 at 11:36

Viewed 239 times

I'm getting error while I'm trying to transfer file from Google cloud storage to google big query. This is the error :

failed with error INVALID_ARGUMENT : Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.

When I look the error I have this :

Error while reading data, error message: JSON parsing error in row starting at position 0: No such field: _ttl.

I don't understand where is the problem. if I just try to transfer 1 files, the data is sent to my Bigquery table. but I need to transfer all the files every days. This is my data format example :

{"createdAt":"2021-12-07T12:07:44.547Z","_lastChangedAt":1638878864561,"isMain":true,"__typename":"Accounting","name":"main","belongTo":"siewecarine","id":"d00ae4ad-c661-40b8-9e90-e0f53b2211fb","_version":1,"updatedAt":"2021-12-07T12:07:44.547Z"}
{"createdAt":"2021-12-07T12:09:12.583Z","_lastChangedAt":1638878952618,"isMain":false,"__typename":"Accounting","name":"test1","belongTo":"mbappe","id":"ee42db80-a6f4-400c-a089-061cc7eec967","_version":1,"updatedAt":"2021-12-07T12:09:12.583Z"}

My data come from a S3 bucket and I batch it from dynamodb.

edited Dec 08 '21 at 11:36

asked Dec 08 '21 at 10:32

Ernest Fokam

Do some files contain _ttl field? – guillaume blaquiere Dec 08 '21 at 12:30
No my files do not content _ttl field – Ernest Fokam Dec 08 '21 at 12:44
Does the table exist? If so, is the _ttl field exit in it? – guillaume blaquiere Dec 08 '21 at 13:12
no I created the table when I look the table schema I don't see it inside – Ernest Fokam Dec 08 '21 at 13:23
I have resolved the issue – Ernest Fokam Dec 09 '21 at 12:08
Any clues for the others that could have the same issue than you? – guillaume blaquiere Dec 09 '21 at 13:36
Hi, please remember to post your solution to help the community. – Kevin Potgieter Dec 16 '21 at 08:15
I changed the data format in s3 when streaming my data from Dynamodb to apache parquet(.parquet) by using a glue crawler and there were others fields that did not appear with Json format. I added those fieds to my table schema in bigquery and everything goes well – Ernest Fokam Dec 17 '21 at 09:18

Getting error while using google data transfer to transfer multiple Json file from GCSto Bigquery

0 Answers0