1

Final goal: transfer csv file contents from google cloud storage bucket to a table in BigQuery dataset.

problem: data transfer seems to be working, not showing any errors/warnings but the data fails to reach the destination table.

  • dataset/table created beforehand
  • bkt created and one csv file uploaded
  • data transfer created and configured to transfer all files from the above bucket to my destination table.
  • transfer is defined On-Demand.

i'm new to bigQuery and GCP but the idea of data transfer seems pretty intuitive to me. i assume i'm missing something here but the flow does not notify me. so i thought it might be permissions (i have bigQuery Admin role among other so..) i thought its a location/region thing but all resources are in EU..

can you help me understand what else am i missing ? attaching some pics that might help.

Bucket Configuration (where my csv resides)

Destination Table details

Transfer Configuration

Data Transfer output showing a successful action

Thanks in advance.

sivan
  • 81
  • 1
  • 1
  • 7
  • 1
    Can you check whether you have minimum 'BigQuery Data Editor' role on your id ? It could be a problem. – Vibhor Gupta Apr 06 '21 at 15:58
  • 1
    Thank you so much for this answer Vibhor. after i updated my roles with BQ Data Editor i managed to make this work. i still dont understand why didnt i get any notification, it seems like a problem/bug to me. anyhow, can you please post this as an answer so i can accept it as the right answer. – sivan Apr 06 '21 at 18:40

2 Answers2

0

Not sure about what's wrong since no error is returned but two points I usually check in that cases:

  • did you try reloading the BigQuery workspace ? It doesn't automatically refresh so you have to reload to see the changes (if any..)

  • have you tried loading the csv directly from the workspace (create Table from Cloud storage in the UI), without setting the Data transfer ? This way you can check straight away if you have an issue with your file

Cylldby
  • 1,783
  • 1
  • 4
  • 17
  • 1
    Thanks for your answer :) . i did try reloading which did not work. i would have tried your suggestion about loading directly but the comment above about "BigQuery Data Editor" role solved it. – sivan Apr 06 '21 at 18:47
0

please have minimum 'BigQuery Data Editor' role on your id to solve your problem.

Vibhor Gupta
  • 670
  • 7
  • 16