How to get the runTime of an active task according to its GMT lanuchTime in Spark?

Question

I want to get the runtime of an active task. In the REST API page of /applications/[app-id]/stages/[stage-id], I can get tasks info in detail.

enter image description here

You can see, the executorRunTime is 0 when a task is not completed. I think I can get the runTime according to launchTime. Suppose the launchTime is 2017-12-21T03:15:31.106GMT. I use the following code to compute the runTime.

val format = new SimpleDateFormat("yyyy-MM-dd'T'hh:mm:ss'.'sss'GMT'", Locale.ENGLISH)
format.setTimeZone(TimeZone.getTimeZone("GMT"))
val launchTime = format.parse("2017-12-21T03:15:31.106GMT").getTime
val runTime = Calendar.getInstance(TimeZone.getTimeZone("GMT")).getTimeInMillis - 
              launchTime

But I get a negative number. Is the time format wrong? Or what?

Answer 1

TL;DR

    String launchTimeString = "2017-12-21T03:15:31.106GMT";
    launchTimeString = launchTimeString.replaceFirst("GMT$", "Z");
    long launchTime = Instant.parse(launchTimeString).toEpochMilli();
    long runTime = System.currentTimeMillis() - launchTime;

(Sorry, I can write only Java 8 code, I will have to trust you to translate.)

java.time and ISO 8601

The date and time classes from Java 1.0 and 1.1 are long outdated, and SimpleDateFormat in particular notoriously troublesome. I recommend you stop using them and use java.time, the modern Java date and time API instead. It is so much nicer to work with.

Your launch time string is a bit funny. It resembles the ISO 8601 standard format with the characteristic T between the date and the time, but has a non-standard GMT in the end, where strings following the standard would usually have Z (or a positive or negative offset from UTC). Since the modern date and time classes parse ISO 8601 as their default, that is, without any explicit formatter, and since writing format pattern strings seems to be an endless source of bugs (it certainly isn’t just you), I found it tempting to modify your string to fit the standard and then parse it.

What was wrong in your format pattern?

There are two bugs in you format pattern string:

• You want uppercase HH for hour of day. Lowercase hh is for hour within AM or PM, in the interval 1 through 12. With SimpleDateFormat this bug usually “just” means that an hour of 12 is understood as 00 (which would have given you a very long run time) (the modern DateTimeFormatter more eagerly tells you you have a bug if you try the same with that class).
• While lowercase ss is correct for seconds, milliseconds are uppercase SSS. This must have been what hit you: 106 in you string was taken to be seconds rather than milliseconds, so if running your code before 03:16:46, you got a negative run time.

So both bugs boil down to: Format pattern strings are case sensitive, so you need to beware the correct case.

Links

• Oracle tutorial: Date Time, explaining how to use java.time.
• Wikipedia article on ISO 8601.