1

I have issues using the newest version of databricks-connect (13.3.0). I would like to access the sparkContext and tried it as it worked for databricks-connect<13.0:

from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.getOrCreate()
spark.sparkContext

However, now I get the error:

pyspark.errors.exceptions.base.PySparkNotImplementedError: [NOT_IMPLEMENTED] sparkContext() is not implemented.

Can someone help?

thebluephantom
  • 16,458
  • 8
  • 40
  • 83
Lazloo Xp
  • 858
  • 1
  • 11
  • 36

1 Answers1

0

Databricks connect in versions 13+ is based on Spark Connect that doesn't support RDD APIs together with related objects like SparkContext. It's really documented as known limitation.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132