0

I cannot access the web UI of my containerized spark cluster even if I copy and paste the following Ip address:

Stopped Spark web UI at http://987c8b219d32:4040

The cluster that I've built is taken from this tutorial Spark Cluster Tutorial

callme_fantastique
  • 11
  • 3

1 Answers1

0

The log specifies that the web UI is stopped. You can't access the Web UI after the spark application have stopped as it is explained in the documentation:

Note that this information is only available for the duration of the application by default.

To enable accessing the web UI after the application have stopped you need to:

set spark.eventLog.enabled to true before starting the application. This configures Spark to log Spark events that encode the information displayed in the UI to persisted storage.

Please check the documentation here.

Also note, that in the tutorial, the default web UI ports are changed to:

SPARK_MASTER_WEBUI_PORT=8080 \
SPARK_WORKER_WEBUI_PORT=8080 \

and that they are exposed outside the container in the docker-compose file to the ports 9090, 9091 and 9092:

  - "9090:8080"
  - "9091:8080"
  - "9092:8080"

Which means that you can access the Web UI using: http://localhost:9090

Zied Yazidi
  • 385
  • 3
  • 9
  • Thanks for the explanation. But the web UI at localhost:9090 doesn't have the stages,jobs,environment...tabs. I need those tabs for monitoring the workload. Following this discussion [link] (https://stackoverflow.com/questions/56397738/sparkui-not-showing-tab-jobs-stages-storage-environment-when-run-in-sta) don't work for me because when I click I get the page not found due to this wrong ip address associated to such link: http://987c8b219d32:4040/ but if I switch it to 9090 as suggested before it return the main page, I would like to know the path to paste in order to access to such infos. – callme_fantastique Jan 03 '23 at 17:33
  • The stages, jobs ... won't appear unless your jobs is still running or the history server is enabled. – Zied Yazidi Jan 03 '23 at 18:18