1

I have a cluster running YARN on it. It has 3 datanodes and 1 client node. I submit all my jobs on the client node. How can I get the elapsed time for all the tasks in a particular job.

Probably RESTful API (https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html) can be used for this purpose. But I am curious to know whether there is any Java API to do the same.

I am able to find the start time for all the task using the method getStartTime() of the TaskReport class. Although the nodes in clusters have times synced using NTP, I don't think it would be a good practice to use the client system current time (System.currentTimeMillis()) to calculate the elapsed time for the Running tasks there can be some accepted lag associated with all the nodes in a cluster even in NTP.

New Coder
  • 499
  • 4
  • 22
  • Why does clock skew matter if you just compute `FinishTime - StartTime`, it's measured on the same host. – Thomas Jungblut Jul 19 '15 at 09:34
  • Yes, agree with that. But the reason why skew matters is that, I want to know the elapsed time for the currently running tasks as well. In case of currently running tasks, the getFinishTime() will return 'Wed Dec 31 19:00:00 EST 1969' as the default value. – New Coder Jul 19 '15 at 09:48
  • That's not really possible I'm afraid. The best you can do is indeed to use the client clock. – Thomas Jungblut Jul 19 '15 at 09:51

1 Answers1

1

In the Job class there is a method called #getTaskReports.

You could use it that way to retrieve the map task duration:

Job job = ...;
job.waitForCompletion(); 

TaskReport[] reports = job.getTaskReports(TaskType.MAP);
for(TaskReport report : reports) { 
   long time = report.getFinishTime() - report.getStartTime();
   System.out.println(report.getTaskId() + " took " + time + " millis!");
}
Thomas Jungblut
  • 20,854
  • 6
  • 68
  • 91
  • As mentioned in the comments for question clarification, this might not work if the Task is still in running state. The finish tIme for which will be 'Wed Dec 31 19:00:00 EST 1969'. – New Coder Jul 19 '15 at 09:52