Versions that I'm using: sparkmonitor 4.1.0 jupyterlab: 4.0.4 spark: 3.4.1
I've done all the steps suggested by https://github.com/itsjafer/jupyterlab-sparkmonitor#readme
The error is:
Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. org.apache.spark.SparkException: Exception when registering SparkListener
After some research, I've found what some say could be a version conflict, others state network, or firewall issues.
This is my config:
spark = SparkSession.builder.appName(f"JupyterNotebook-{user_name}")\
.config("spark.extraListeners", "sparkmonitor.listener.JupyterSparkMonitorListener")\
.config("spark.driver.extraClassPath", "/opt/conda/lib/python3.11/site-packages/sparkmonitor/listener.jar")\
.getOrCreate()
the listener.jar file exists in that directory, I've checked it.
just a reminder. I'm using jupyterhub that spawns containered jupyterlab images.
