0

We are trying to run Hive queries on HDP 2.1 using GCS Connector, it was working fine until yesterday but since today morning our jobs are randomly started failing. When we restart them manually they just work fine. I suspect it's something to do with number of parallel Hive jobs running at a given point of time.

Below is the error message:

vertexId=vertex_1407434664593_37527_2_00, diagnostics=[Vertex Input: audience_history initializer failed., java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found] DAG failed due to vertex failure. failedVertices:1 killedVertices:0 FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask

Any help will be highly appreciated.

Thanks!

Igor Dvorzhak
  • 4,360
  • 3
  • 17
  • 31
user3870452
  • 33
  • 4
  • Thanks for the report, just a few questions about your setup: Is the error still happening? When it happens, are you able to type "hadoop fs -ls gs:///" to classload the GoogleHadoopFileSystem class? It sounds like this occurred on a long-running cluster; where there any other changes to the cluster at the time they started failing? Is the cluster running on your own hardware, or is it on Google Compute Engine? – Dennis Huo Sep 22 '14 at 23:15
  • 1
    This issue is fixed, I have placed gcs hadoop connector under /apps/tez/lib and restarted the hive services. – user3870452 Sep 24 '14 at 21:54

0 Answers0