3

<master_IP>:8081 The image shows 8081 UI. The master shows running application when I start a scala shell or pyspark shell. But when I use spark-submit to run a python script, master doesn't show any running application. This is the command I used: spark-submit --master spark://localhost:7077 sample_map.py. The web UI is at :4040. I want to know if I'm doing it the right way for submitting scripts or if spark-submit never really shows running application.

localhost:8080 or <master_ip>:8080 doesn't open for me but <master_ip>:8081 opens. It shows the executor info.

These are my configurations in spark-env.sh:

export SPARK_EXECUTOR_MEMORY=512m 
export SPARK_MASTER_WEBUI_PORT=4040
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=1g
export SPARK_WORKER_INSTANCES=2 
export SPARK_WORKER_DIR=/opt/worker
export SPARK_DAEMON_MEMORY=512m
export SPARK_LOCAL_DIRS=/tmp/spark  
export SPARK_MASTER_IP 'splunk_dep'

I'm using CentOS , python 2.7 and spark-2.0.2-bin-hadoop2.7.

kavya
  • 759
  • 4
  • 14
  • 31

4 Answers4

3

You can open spark master’s web UI, which is http://localhost:8080 by default to see running apps (in standalone cluster mode) :Spark Master UI

If multiple apps are running - they will bind to port 4040, 4041, 4042 ...

You can access this interface by simply opening http://:4040 in a web browser. If multiple SparkContexts are running on the same host, they will bind to successive ports beginning with 4040 (4041, 4042, etc).

Artur Sukhenko
  • 602
  • 3
  • 12
  • Yes. But the same doesn't work only for spark-submit using a python script. The pic that you've shown is for the scala shell right? Scala shell works for me also on 4040. – kavya Dec 01 '16 at 12:18
  • It does. I've ran ./bin/spark-submit --master spark://node7:7077 examples/src/main/python/pi.py 1000 and it did showup in UI on 4040 and in completed apps in 8080. Are you sure spark-master is running? Try ./sbin/start-all.sh – Artur Sukhenko Dec 01 '16 at 12:59
  • Did you run spark shell in local mode or standalone cluster? Here is documentation on how to setup standalone cluster mode http://spark.apache.org/docs/latest/spark-standalone.html – Artur Sukhenko Dec 01 '16 at 13:09
  • @kaks What does 8081 say Master URL is? – Artur Sukhenko Dec 01 '16 at 13:22
  • I've attached the image in the question. – kavya Dec 01 '16 at 13:38
  • And I've done the clustering in standalone mode I assume because that was the doc that i followed. The spark UI shows that the master is still ALIVE. Could you tell me what is `1000` you've mentioned in your spark-submit command? – kavya Dec 01 '16 at 13:39
  • That is an argument for pi.py. Can you run this: ./bin/spark-submit --master spark://splunk_dep:7077 sample_map.py – Artur Sukhenko Dec 01 '16 at 14:31
  • That also doesn't work for me. I don't get it. It works for pyspark and scala shell but not spark-submit. – kavya Dec 02 '16 at 05:22
  • Thanks. It turned out that the port is on 4041, not 4040 – Chau Pham Dec 03 '18 at 02:01
0

Are you accessing SPARK-UI when the application is running or after it completed its execution?

Try adding some code, which will wait for key-press (hence the spark execution won't end) - and see if it solve your problem.

Yaron
  • 10,166
  • 9
  • 45
  • 65
  • I keep refreshing UI while the script is running and after it has completed. But still it doesn't show. I tried adding user input in between, even then there is no change. – kavya Dec 01 '16 at 13:07
0

You just go to localhost:8080 and check that there is one completed application which you submit.

Sandeep Purohit
  • 3,652
  • 18
  • 22
  • It doesn't show any completed application in 4040 for the spark-submit alone. For me 8080 is not reachable. – kavya Dec 01 '16 at 13:06
0

For Local run use this:

val sparkConf = new SparkConf().setAppName("Your app Name").setMaster("local")
val sc = new SparkContext(sparkConf)

while you do sparkSubmit:

val sparkConf = new SparkConf().setAppName("Your app Name")
val sc = new SparkContext(sparkConf)

This won't work in local test but when you compile with this and spark submit job it will show in UI.

Hope this explains.

toofrellik
  • 1,277
  • 4
  • 15
  • 39
  • I haven't used that. But I did initialize SparkContext like `sc = SparkContext("local","test")`. If I change it to `sc = SparkContext("localhost","test")`, I get a parsing error : ` Could not parse Master URL: 'localhost'` – kavya Dec 02 '16 at 10:30
  • if you are running on master you dont need to specify any, remove local and run it it will show up. – toofrellik Dec 02 '16 at 11:16
  • `val sparkConf = new SparkConf().setAppName("Ad Request Metrics").setMaster("local")` use this, here for test purpose use`.setMaster("local")`. but when ever you are running in master remove that – toofrellik Dec 02 '16 at 11:18
  • Updated my answer check that out – toofrellik Dec 02 '16 at 11:35
  • I'm using python. So i tried `sparkConf = new SparkConf().setAppName("test") sc = SparkContext(sparkConf)` and I'm getting invalid syntax for `SaprkConf()`. Have I written it right? – kavya Dec 04 '16 at 16:24
  • Thank you so much! After so much struggle got it right. In pyspark I defined it like this: `sparkConf = SparkConf().setAppName("test") sc = SparkContext(conf=sparkConf)`. Now the UI is showing me the running application. – kavya Dec 05 '16 at 07:17