I would like to connect R instance to Spark running on HDP-Sandbox deployed on Docker on one local machine. The error message indicates that --version call on spark-submit fails.
R instance
packageVersion("sparklyr")
# [1] ‘1.0.1’
# Set old JAVA version
Sys.setenv("JAVA_HOME" =
"/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home")
library(sparklyr)
sc <- spark_connect(
master = " http://localhost:4040",
spark_home = "/usr/hdp/current/spark2-client",
app_name = "RStudio",
method = "shell"
)
Error in
system2(file.path(spark_home, "bin", "spark-submit"), "--version",: error in running command
HDP
[root@sandbox-hdp ~]# /usr/hdp/current/spark2-client/bin/spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.1.3.0.1.0-187
/_/
Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_191
Branch HEAD
Compiled by user jenkins on 2018-09-19T10:10:07Z
Revision fe7bed1ca174a6687ebd2aa0f8ba5fb7bf668399
Url git@github.com:hortonworks/spark2.git
Type --help for more information.
Update
I've resorted to using Livy as it seems to be an easier solution. I have opened 8999 port that is referenced in the livy.conf file and can now access Livy UI at http://localhost:8999/ui.
Despite this, I still keep getting errors when trying to connect:
config <- livy_config()
sc <- spark_connect(master = "http://localhost:8999",
method = "livy",
config = config)
# Error in livy_connection(master = master, config = config, app_name, version, :
# Failed to launch livy session, session status is still starting after waiting for 60 seconds
config <- livy_config(username = "root", password = "***")
sc <- spark_connect(master = "http://localhost:8999",
method = "livy",
config = config)
# Error in livy_connection(master = master, config = config, app_name, version, :
# Failed to launch livy session, session status is still starting after waiting for 60 seconds
