0

I have added PYTHONPATH with the path to the Spark libraries in an IntelliJ project:

enter image description here

However the PySpark classes are not accessible:

enter image description here enter image description here

So what further needs to be done to the IJ project?

zero323
  • 322,348
  • 103
  • 959
  • 935
WestCoastProjects
  • 58,982
  • 91
  • 316
  • 560

1 Answers1

2

You should add PySpark libraries to your project path as well. Assuming you're using IDEA Intellij, not PyCharm go to File -> Project Structure -> SDKs and configure Classpath for Python interpreter.

zero323
  • 322,348
  • 103
  • 959
  • 935
  • This is a mixed project with scala, java, and python: i tried what you said once in the past - thus changing from a java sdk to python. The project got corrupted and I had to rebuild it. Do you have comment on how to work with the mixed languages project? – WestCoastProjects Dec 25 '15 at 18:57
  • It appears that I should create a separate python-only module. I'll upvote for kind of pointing the general direction – WestCoastProjects Dec 25 '15 at 19:02
  • Separate project would be my personal preference (especially with PyCharm which works much better than IDEA) but if you use modules in your project you can define module specific SDK. – zero323 Dec 25 '15 at 19:09
  • A year later and I can not reproduce the steps to make this work. Have you tried recently? In particular I had tried adding `pyspark.zip` and `py4j..jar` but get `Exception: Java gateway process exited before sending the driver its port number` – WestCoastProjects Feb 26 '17 at 01:53
  • I don't use intelij on daily basis but as far as I remember I used PySpark source root (`$SPARK_HOME/python`). Though gateway exit may suggest some other problem. If you don't resolve it ping me with the details (Spark version, Python version, InteliJ version). – zero323 Feb 26 '17 at 02:06
  • I'll make another question - you'll deserve getting the points. – WestCoastProjects Feb 26 '17 at 02:12
  • I checked, and it still works for me (Spark 2.1, IDEA 2016.3.4). – zero323 Feb 26 '17 at 21:48