CDH5.4.2 spark can use HiveContent in spark-shell but can't open spark-sql

Question

I am using spark(standalone) of CDH5.4.2

After copying hive-site.xml to $SPARK_HOME/conf,i can query from hive in spark-shell,such as below:

scala> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc); hiveContext: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@6c6f3a15 scala> hiveContext.sql("show tables").show();

But when i open spark-sql ,it show wrong:

java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver Failed to load main class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.

You need to build Spark with -Phive and -Phive-thriftserver.

What is different between spark-shell and spark-sql? If spark of cdh don't support hive,why can i use HiveConext?

What exactly do you think the `spark-sql` command is supposed to do?!? — Samson Scharfrichter, Jul 26 '16 at 10:59
Spoiler: Cloudera explicitly states that the Spark **JDBC Thrift server** is not supported in CDH, so it's not surprising that you cannot start it. — Samson Scharfrichter, Jul 26 '16 at 11:00
Thank you so much ,I will carefully to read the official document. — 陈志明, Jul 26 '16 at 16:59

score 0 · Answer 1 · answered Feb 13 '21 at 18:55

Cloudera has a list of unsupported features here:

https://docs.cloudera.com/runtime/7.2.6/spark-overview/topics/spark-unsupported-features.html

The Thrift server is not supported.

This is a copy of the list for 7.2.6:

Apache Spark experimental features/APIs are not supported unless stated otherwise.
Using the JDBC Datasource API to access Hive or Impala is not supported
ADLS not supported for All Spark Components. Microsoft Azure Data Lake Store (ADLS) is a cloud-based filesystem that you can access through Spark applications. Spark with Kudu is not currently supported for ADLS data. (Hive on Spark is available for ADLS.)
IPython / Jupyter notebooks is not supported. The IPython notebook system (renamed to Jupyter as of IPython 4.0) is not supported.
Certain Spark Streaming features, such as the mapWithState method, are not supported.
Thrift JDBC/ODBC server is not supported
Spark SQL CLI is not supported
GraphX is not supported
SparkR is not supported
Structured Streaming is supported, but the following features of it are not:
Continuous processing, which is still experimental, is not supported.
Stream static joins with HBase have not been tested and therefore are not supported.
Spark cost-based optimizer (CBO) not supported.

CDH5.4.2 spark can use HiveContent in spark-shell but can't open spark-sql

1 Answers1