8

I am new to Spark and Python. I have installed python 3.5.1 and Spark-1.6.0-bin-hadoop2.4 on windows.

I am getting the below error when I execute sc = SparkContext("local", "Simple App") from the Python shell:

>>> from pyspark import SparkConf, SparkContext

>>> sc = SparkContext("local", "Simple App")

Traceback (most recent call last):

  File "<pyshell#11>", line 1, in <module>

    sc = SparkContext("local", "Simple App")

  File "C:\spark-1.6.0-bin-hadoop2.4\python\pyspark\context.py", line 112, in __init__

    SparkContext._ensure_initialized(self, gateway=gateway)

  File "C:\spark-1.6.0-bin-hadoop2.4\python\pyspark\context.py", line 245, in _ensure_initialized

    SparkContext._gateway = gateway or launch_gateway()

  File "C:\spark-1.6.0-bin-hadoop2.4\python\pyspark\java_gateway.py", line 79, in launch_gateway

    proc = Popen(command, stdin=PIPE, env=env)

  File "C:\Python35-32\lib\subprocess.py", line 950, in __init__
    restore_signals, start_new_session)

  File "C:\Python35-32\lib\subprocess.py", line 1220, in _execute_child
    startupinfo)

FileNotFoundError: [WinError 2] The system cannot find the file specified

>>>
miken32
  • 42,008
  • 16
  • 111
  • 154
sam
  • 101
  • 1
  • 1
  • 6

5 Answers5

4

Check your address to make sure it is written correctly. In my case, I had the address as:

"C:/Users/nekooeimehr/AppData/Local/Programs/Python/Python35-32/spark-1.6.2-bin-hadoop2.4"

while the correct address is:

"C:/Users/nekooeimehr/AppData/Local/Programs/Python/Python35-32/spark-1.6.2-bin-hadoop2.4/spark-1.6.2-bin-hadoop2.4"

m00am
  • 5,910
  • 11
  • 53
  • 69
Iman Nekooeimehr
  • 101
  • 4
2

You have to set the SPARK_HOME correctly. I have debugged the python scripts and verified. This would work.

bharathi
  • 41
  • 1
2

Restart and Run pySpark as an administrator

paul
  • 519
  • 6
  • 13
0

If the verified the system environment variables and it is still not working, check if your jvm (64 or 32)bit version is compatible with your machine.

SureshCS
  • 1,005
  • 12
  • 23
0

I had the same error. Make sure SPARK_HOME, JAVA_HOME and the PATH environment variables set correctly

For my machine,

SPARK_HOME: C:\Repo\spark\spark-3.3.1-bin-hadoop3
JAVA_HOME: C:\Program Files\Java\jdk1.8.0_361

In your PATH variable: Append the following

%SPARK_HOME%\bin
%JAVA_HOME%\bin

This is because the SPARK_HOME\bin contains the executables for spark-shell/sql etc.

deerishi
  • 527
  • 7
  • 5