0

I am following this WordCount example using the Google BigQuery-Hadoop connector:

https://developers.google.com/hadoop/writing-with-bigquery-connector#completecode

The example works fine as it is.

To test array in the output schema, I have altered just one line in the code by adding an array object definition to the output schema:

String outputTableSchema = "[{'name': 'Word','type': 'STRING'},{'name': 'Number','type': 'INTEGER'},{'name':'Persons','mode':'REPEATED','type':'RECORD','fields':[{'name': 'name','type': 'STRING'},{'name': 'age','type': 'INTEGER'}]}]";

Now when I run the WordCount example, it gives this exception:

java.lang.IllegalStateException at com.google.gson.JsonArray.getAsString(JsonArray.java:133) at com.google.cloud.hadoop.io.bigquery.BigQueryUtils.getSchemaFromString(BigQueryUtils.java:97) at com.google.cloud.hadoop.io.bigquery.BigQueryOutputFormat.getRecordWriter(BigQueryOutputFormat.java:121) at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:568) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.mapred.Child.main(Child.java:249)

Does anyone know what the issue is?

Thank you

Felipe Hoffa
  • 54,922
  • 16
  • 151
  • 325
user3709284
  • 323
  • 1
  • 2
  • 10

1 Answers1

1

This is actually a bug in the current version of the BigQuery connector which prevents it from supporting inner records with more than 1 field.

We have a fix internally and it's slated to go out with the next release (0.4.3) which may still be a couple weeks out; if you'd like to help try out a staging build, feel free to reach out to gcp-hadoop-contact@google.com and we can provide instructions.

Dennis Huo
  • 10,517
  • 27
  • 43