0

I am using pyspark to run an application on a cluster in client mode using standalone for monitoring.

All I want to do is see the logs.

I've tried two things:

1) I went to the config file (spark-defaults.conf) in SPARK_HOME: 

spark.eventLog.dir     hdfs:<ip>:<port>/<path to directory/

2) set in my python script the following:

conf = pyspark.SparkConf()
conf.set("spark.eventLog.dir", 'hdfs:<ip>:<port>/<path to directory/')
sc = pyspark.SparkContext(conf=conf)

Neither of these seem to produce logs in the folder. Is there anything else that I can try?

Thank you. this is spark 1.3

makansij
  • 9,303
  • 37
  • 105
  • 183
  • 1
    do you want spark logs or spark event logs ? – Knight71 Aug 04 '16 at 06:28
  • oy. what's the difference? – makansij Aug 04 '16 at 06:42
  • 1
    http://spark.apache.org/docs/latest/monitoring.html - spark event logs is to view webui after your application had run, basically it contains the web ui related metrics. spark logs is the actual application logs. – Knight71 Aug 04 '16 at 06:45
  • based on what i've read - spark logs. Thank you @Knight71 – makansij Aug 04 '16 at 20:43
  • If you only want to view the logs, you can head over to the history server - `http://:18080/api/v1` and check under`/applications/[base-app-id]/logs` – KartikKannapur Aug 08 '16 at 08:11
  • That only works if you are using `YARN` or `MESOS`. It doesn't work for standalone. The link provided by @Knight71 provided even says that http://spark.apache.org/docs/latest/monitoring.html "If Spark is run on Mesos or YARN,..." – makansij Aug 21 '16 at 00:14

0 Answers0