1

I have Alluxio 1.8 installed on an EMR 5.19.0 cluster, and can see my S3 tables using /usr/local/alluxio/bin/alluxio fs ls /.

However, when I start up hive and issue hive> [[DDL w/ LOCATION = alluxio://master_host:19998/my_table ]]], I get the following:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.RuntimeException: java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found

Is there a way of getting past this? I've tried starting hive with --auxpath pointing to both /usr/local/alluxio/client/alluxio-1.8.1-client.jar and a copy of the jar on hdfs without any success.

Any help?

rongenre
  • 1,334
  • 11
  • 21

1 Answers1

1

I posted a blog talking about the reasons for the error message java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found. Here are some tips, hope they can help:

  • For Hive, set environment variable HIVE_AUX_JARS_PATH in conf/hive-env.sh:
export HIVE_AUX_JARS_PATH=/<PATH_TO_ALLUXIO>/client/alluxio-1.8.1-client.jar:${HIVE_AUX_JARS_PATH}

which I guess is equivalent to what you have done to set --auxpath.

  • Depending on your setting of Hive (e.g., Hive on MR or Spark or Tez), you may also need to make sure the runtime is also able to access the client jar. Take Hive on MR as an example, you perhaps also need to append the path to Alluxio client jar to mapreduce.application.classpath or yarn.application.classpath to ensure each task of the MR jobs can access this jar.
apc999
  • 250
  • 3
  • 6
  • I'm able make progress by setting `HIVE_AUX_JARS_PATH` to `/usr/local/alluxio/client/alluxio-1.8.1-client.jar` by configuring hive-env in the EMR configuration. I'm still not able to create the temporary table though, I'm getting ` FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: alluxio.exception.status.UnavailableException Failed to connect to FileSystemMasterClient @ :19998 after 44 attempts) ` – rongenre Dec 27 '18 at 03:13
  • @rongenre, can you check if the solution of this issue helps? https://stackoverflow.com/questions/52810985/plain-authentication-failed-user-yarn-is-not-configured-for-any-impersonation – apc999 Dec 29 '18 at 07:13
  • Impersonation took care of part of it, but I'm running into further downstream problems. Will update when I get a resolution. – rongenre Jan 02 '19 at 20:01