1

I am trying to run Sparkling Water on my Local instance of Spark 2.1.0. I followed documentation on H2o for Sparling Water. But when I try to execute 

sparkling-shell.cmd

I am getting following error :

The filename, directory name, or volume label syntax is incorrect.

I look into the batch file and I am getting this error when the following command is executed:

C:\Users\Mansoor\libs\spark\spark-2.1.0/bin/spark-shell.cmd --jars C:\Users\Mansoor\libs\H2o\sparkling\bin\../assembly/build/libs/sparkling-water-assembly_2.11-2.1.0-all.jar --driver-memory 3G --conf spark.driver.extraJavaOptions="-XX:MaxPermSize=384m"

When I remove --conf spark.driver.extraJavaOptions="-XX:MaxPermSize=384m", Spark starts but I am unable to import the packages of H2o.

import org.apache.spark.h2o._

error: object h2o is not a member of package org.apache.spark

I tried everything I could but unable to solve this issue. Could someone help me in this? Thanks

Mansoor
  • 1,157
  • 10
  • 29

2 Answers2

1

Please try to correct your path:

C:\Users\Mansoor\libs\spark\spark-2.1.0/bin/spark-shell.cmd --jars C:\Users\Mansoor\libs\H2o\sparkling\bin\..\assembly\build\libs\sparkling-water-assembly_2.11-2.1.0-all.jar --driver-memory 3G --conf spark.driver.extraJavaOptions="-XX:MaxPermSize=384m"

There is also doc page about RSparkling at Windows, which can contain different troubleshooting tips... https://github.com/h2oai/sales-engineering/tree/master/megan/RSparklingAndWindows

Michal
  • 437
  • 3
  • 8
1

Problem is with spark-shell command while submitting jars. Workaround is to modify spark-defaults.conf

Adding spark.driver.extraClassPath and spark.executor.extraClassPath parameters to spark-defaults.conf file as follows:

spark.driver.extraClassPath    \path\to\jar\sparkling-water-assembly_version>-all.jar

spark.executor.extraClassPath   \path\to\jar\sparkling-water-assembly_version>-all.jar

And Remove --jars \path\to\jar\sparkling-water-assembly_version>-all.jar from sparkling-shell2.cmd

Mansoor
  • 1,157
  • 10
  • 29