1

I am having trouble running queries with Shark locally on Mac OSX 10.8. I am trying to run some test queries on data stored in Hive. I am using Scala 2.9.3 and Hive 0.9.0 and both seem to be running fine. The Hive database is using MySQL to store metadata.

I can start up Shark fine. I can see all my databases in Hive in the Shark console. I can switch to the desired database. But when I try to query data I get errors.

The simple query looks like this:

select * from table limit 100;

The error looks like this:

25.351: [Full GC 98320K->17564K(1013632K), 0.1279630 secs]
java.lang.NoSuchMethodError: org.apache.hadoop.hive.shims.HadoopShims.isLocalMode(Lorg/apache/hadoop/conf/Configuration;)Z
    at shark.execution.FileSinkOperator.execute(FileSinkOperator.scala:123)
    at shark.execution.SparkTask.execute(SparkTask.scala:101)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:134)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1326)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1118)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
    at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:294)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341)
    at shark.SharkCliDriver$.main(SharkCliDriver.scala:203)
    at shark.SharkCliDriver.main(SharkCliDriver.scala)
FAILED: Execution Error, return code -101 from shark.execution.SparkTask
ChrisGPT was on strike
  • 127,765
  • 105
  • 273
  • 257
DJElbow
  • 3,345
  • 11
  • 41
  • 52

1 Answers1

0

this looks like a recent addition to Shark code, https://github.com/amplab/shark/pull/140/files perhaps wasn't tested with Hive 0.9?

current HadoopShims.java does have the isLocalMode(), but the Hive 0.9 HadoopShims.java does not have the isLocalMode() method.

I would recommend using at least Hive 0.10, or even better a recent 0.12 if you can.

Remus Rusanu
  • 288,378
  • 40
  • 442
  • 569
  • Well, when downloading Shark, it actually ships with a version of Hive which is 0.9.0. Using this version hasn't worked either. https://github.com/amplab/shark/wiki/Running-Shark-Locally – DJElbow Feb 03 '14 at 20:03
  • Try adding `-verbose:class` and then analyze where is the shim loaded from. Perhaps a wrong CLASSPATH? I reckon I never run Shark myself. – Remus Rusanu Feb 03 '14 at 20:08
  • Looks like the version was the issue. Hive 0.9.0 from here http://archive.apache.org/dist/hive/hive-0.9.0/ was not working. However, using the 0.9.0 version that ships with Shark does work. Now that I am using the version of Hive that shipped with Shark and have the mysql connector in place my queries are working. – DJElbow Feb 03 '14 at 21:10