6

I create a pyspark dataframe and i want to see it in the SciView tab in PyCharm when i debug my code (like I used to do when i have worked with pandas). It says "Nothing to show" (the dataframe exists, I can see it when I use the show() command).

someone knows how to do it or maybe there is no integration between pycharm and pyspark dataframe in this case?

nofar mishraki
  • 526
  • 1
  • 4
  • 15
  • 1
    Beware that any such action if possible will mean collecting data from all executor into driver memory...which for large data size may result into crash...show/head etc. are safer alternative to just show sample data – abhiieor Dec 05 '18 at 08:18

1 Answers1

7

Pycharm does not support spark dataframes, you should call the toPandas() method on the dataframe. As @abhiieor mentioned in a comment, be aware that you can potentially collect a lot of data, you should first limit() the number of rows returned.

Manu Valdés
  • 2,343
  • 1
  • 19
  • 28
  • https://youtrack.jetbrains.com/issue/BDIDE-2399/ feature request for the support in Big Data Tools plugin. – maciek Aug 30 '23 at 20:17