0

I'm currently working on a BI stack that flows from bigquery to Tableau. I'm trying to use Dataprep to remove unecessary columns and join the tables in bigquery to create a "master" table to then feed into Tableau. the tables in bigquery update about every 8 hours and I'm trying to create a similar schedule in dataprep for the master table.

Running a job or even trying to take a sample from the joined tables keeps producing this error.

java.lang.IllegalStateException: The schema of the BigQuery table does not match the recipe.

Not sure how I'm supposed to troubleshoot this. I compared the two columns in the error (expected vs actual columns) and found it was expecting one more extra column. Why would this happen?

For comparison, I have a postgres db running in parallel with the same data and am able to make the same joins with no issues.

Also, here is how the overall flow looks in dataprep.

Any help is appreciated.

Thanks,

Art

Artin Sarkisian
  • 1
  • 1
  • Can you check in BigQuery UI the columns in your destination table and compare it to the error (I also see an extra float field at the end of the list). BigQuery probably enforce different restrictions on your table than postgress which is why you see the error – Tamir Klein Dec 26 '18 at 20:36
  • Hi Tamir, thanks for the quick response. so the problem occurs even before committing the data to the destination table. I get the same error when trying to run a sample, so not sure if its related to the destination table (also the destinatation table was created when I ran the job). The odd part of this is is that I actually ran it successfully one time, and it started failing on the schedule refreshes. – Artin Sarkisian Dec 26 '18 at 20:56
  • Have you tried to check in BigQuery UI the Query history or the Job History to see if you can get a more detail error message? If you can also try and print the insert record with the data and try to run it directly in BigQuery it might help pinpoint the issue – Tamir Klein Dec 26 '18 at 21:03

2 Answers2

0

Sometimes the data source could change columns (adds or eliminate columns). Re-import data solve this issue.

enter image description here

molavec
  • 8,848
  • 1
  • 27
  • 22
0

Dataprep uses ML algorithm that TRY to help you, but most of times auto-interprets the type and when you do a new recipe, the type of the column is changed. Therefore the last recipe that you have to do is 'change type', in order to be sure that the dataprep columns have the same type than BigQuery columns.

jonaetn
  • 9
  • 4