0

I am trying to performance tune a slow running DSX job.

I have navigated to the spark history server from the underlying spark service on Bluemix (as per this question).

I have executed a cell containing some basic spark code:

In [1]:
x = sc.parallelize(range(1, 1000000))
x.collect()

Out[1]:
[1,
 2,
 3,
 4,
 5,
 ...

I have then refreshed the Job History Server page in the browser, however, the spark history server is not showing any complete applications:

enter image description here

How can I find the 'complete' applications?

Update

The spark service I'm referring to is IBM's managed spark service on Bluemix so I don't have any control over the configuration.

Update 2

It looks as though the dates are getting corrupted which is why I'm not seeing completed jobs:

enter image description here

Community
  • 1
  • 1
Chris Snow
  • 23,813
  • 35
  • 144
  • 309

3 Answers3

1

The notebook context starts a Spark app, which will keep running for 20 minutes after your job completed. When you submit another job during that time, it will use the same app and appear in the same history entry. Therefore, you cannot track job/activity completion of notebooks or Interactive API kernels in the history server.

Spark activity generates output in the kernel log, maybe that can provide the information you're looking for.

Roland Weber
  • 1,865
  • 2
  • 17
  • 27
  • Thanks @Roland. I really need the history server so that I can debug jobs that are not performant. I just checked (4 hours later) and the history server still isn't showing any complete applications. Is it possible to force flushing to the log so I don't have to wait 20 minutes before I can start assessing my job? – Chris Snow Mar 07 '17 at 11:25
  • Sorry, I'm not familiar with that part of the Spark infrastructure. That's SparkEGO resource management. When you restart kernels, you might not get a completion entry in the history server at all. – Roland Weber Mar 07 '17 at 11:30
  • Np - upvoting because the answer will be useful for other users. However, it looks as though my job history log is corrupt. I've updated the question to reflect that. – Chris Snow Mar 07 '17 at 11:37
0

Did you configure your spark cluster to have event log? Like this:

spark.eventLog.enabled true
spark.eventLog.dir hdfs://namenode/shared/spark-logs
Hlib
  • 2,944
  • 6
  • 29
  • 33
0

I have taken this up with the spark service engineering team - it is a known issue.

Chris Snow
  • 23,813
  • 35
  • 144
  • 309