SparkUI not showing Tab (Jobs, Stages, Storage, Environment,...) when run in standalone mode

Question

I'm running spark master through the following command:

./sbin/start-master.sh

After that I went to http://localhost:8080, and I saw the following page.

spark

I was expecting to see the tab with Jobs, Environments, ... like the following

spark

Could someone help me to understand where the problem is?

Do I need extra configuration?

Thanks

Giuseppe

score 2 · Answer 1 · answered Sep 09 '20 at 18:10

2

The UI you get at <master-ip>:8080 is the Masters' Web UI. You can have one similar for Workers at <worker-ip>:8081 (8081 is default port; could be any)

The Web UI you are looking for has specifications to the Application you run i.e. the executors, the dag-diagram, the stages etc. So if you click on one of the Running Applications, you will get a page like below. Click on the Application UI, and you get the required page.

answered Sep 09 '20 at 18:10

khari-sing

650
6
16

agree. the only thing is that for me spark 2.4.3, k8s. worker port is 4040 – rigby Nov 30 '20 at 16:46

Avinash · Answer 2 · 2019-06-03T14:31:35.210

1

To monitor and inspect Spark job executions in a web browser access the port 4040 instead of 8080.

For example : http://ipAddress:4040

Replace 8080 in the URL.

Refer to : https://spark.apache.org/docs/latest/monitoring.html

In the above link, spark documentation states that it launches the web interface in port 4040

One more useful note is this information is only available for the duration of the application by default. To view the web UI after the fact, set spark.eventLog.enabled to true before starting the application. This configures Spark to log Spark events that encode the information displayed in the UI to persisted storage.

Hope that helps.

edited Jun 03 '19 at 14:31

answered Jun 01 '19 at 16:13

Avinash

11
2

1

From Spark documentation looks like the spark ui has a standard port (8080). https://spark.apache.org/docs/1.4.1/spark-standalone.html#monitoring-and-logging There is no mention about 4040 Could you point to me some docs please? – Giuseppe Scopelliti Jun 02 '19 at 13:25
Please refer to https://spark.apache.org/docs/latest/monitoring.html for the reference, and also updated in the above answer. Let me know if you have any queries. :) – Avinash Jun 03 '19 at 14:26

score 0 · Answer 3 · answered Feb 17 '20 at 16:57

0

I think what you are looking for is the SparkUI, which you should first create a SparkContext. For example, you could run pyspark if you have that installed, and when it starts correctly, you would see a sparkUI spawned at port 4040 or 4041...

answered Feb 17 '20 at 16:57

Yicheng Wang

36
3

SparkUI not showing Tab (Jobs, Stages, Storage, Environment,...) when run in standalone mode

3 Answers3

Linked