1

Because i have this issue in my ipython3 notebook, i guess i have to change "spark-env.sh.template" somehow.

Exception: Python in worker has different version 2.7 than that in driver 3.4, PySpark cannot run with different minor versions

bytecode77
  • 14,163
  • 30
  • 110
  • 141
nicolasdavid
  • 2,821
  • 4
  • 18
  • 22

2 Answers2

4

Spark does not yet work with Python 3.If you wish to use the Python API you will also need a Python interpreter (version 2.6 or newer).

I had the same issue when running IPYTHON=1 ./pyspark.

Ok quick fix

Edit vim pyspark and change PYSPARK_DRIVER_PYTHON="ipython" line to 

PYSPARK_DRIVER_PYTHON="ipython2"

That's it.

If you want to check where dose ipython points to,

Type which ipython in terminal and I bet that'll be

/Library/Frameworks/Python.framework/Versions/3.4/bin/ipython

**UPDATED**

The latest version of spark works well with python 3. So this may not need with the latest version.

Just set the environment variable:

export PYSPARK_PYTHON=python3

in case you want this change to be permanent add this line to pyspark script

TMKasun
  • 784
  • 7
  • 24
  • 2
    Spark supports Python 3 now and at the time of the answer as well. See e.g.: http://stackoverflow.com/questions/30279783 – Reid Mar 22 '16 at 23:05
2

I believe you can specify the two separately, like so:

PYSPARK_PYTHON=/opt/anaconda/bin/ipython
PYSPARK_DRIVER_PYTHON=/opt/anaconda/bin/ipython

Based on this other question Apache Spark: How to use pyspark with Python 3.

Community
  • 1
  • 1
Kevin Dahl
  • 752
  • 5
  • 11