0
import databricks.koalas as ks

df = ks.DataFrame({'x': [1, 2], 'y': [3, 4], 'z': [5, 6]})
df.columns = ['x', 'y', 'z1']
df['x2'] = df.x + df.x    
print(df)

Unable to get any output when ran in JupyterNotebook. Code is continuously running with a "WARNING:root:Found pyspark version "2.3.0" installed. pyspark>=2.4.0 is recommended."

Following are the libraries used.

pandas- 0.25
PySpark- 2.3
pyarrow- 0.13
matplotlib - 3.1
databricks - 0.2
koalas - 0.21
Jaroslav Bezděk
  • 6,967
  • 6
  • 29
  • 46
gayathri nadella_user6699670
  • 21
  • 2
  • 6
  • as the warning suggests, have you tried updating PySpark? – CAPSLOCK Nov 04 '19 at 10:02
  • We have a client server setup with Spark 2.3 version. Do not want to change that as other applications are running on it. It was working in my colleague's laptop despite of the warning. – gayathri nadella_user6699670 Nov 04 '19 at 10:34
  • Possible solution might be to create a virtual environment with an updated PySpark version. Otherwise, do you only encounter the problem on JupyterNotebook?. Could you please provide the version of Jupyter lab and Jupyter notebook you are using? I assume your colleague has the same versions? – CAPSLOCK Nov 04 '19 at 10:50

0 Answers0