0

I am trying to run graphframes in pyspark (in Ubuntu) and followed the below steps:

I edited mu .profile file like below :

SPARK_PATH=/home/spark/spark-2.4.4-bin-hadoop2.7
# set PATH so it includes user's private bin directories
PATH="$HOME/bin:$HOME/.local/bin:$PATH"
$SPARK_PATH/bin/pyspark --master local[2]
export PYSPARK_DRIVER_PYTHON="jupyter"

and then I used to open pyspark by below command 

pyspark --packages graphframes:graphframes:0.7.0-spark2.4-s_2.11

through terminal .But instead of opening jupyter notebook it open the spark session in the terminal.Not sure which step is not making it open through jupyter.

Any help is appreciated.

Ricky
  • 2,662
  • 5
  • 25
  • 57

1 Answers1

0

in order that the notebook is started from pyspark you have to two environment variables

export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook
Alex Ortner
  • 1,097
  • 8
  • 24