3

I have an server setup that consists of 8 machines on which I run a Hadoop job to download certain assets. A client side agent uses JobConf to start the Hadoop jobs.

The agent gets a response code from the server based on which it can determine how the download proceeded. In the below code, invoke() creates a connection to the REST API exposed by the server. ResponseData is a custom class that can read the downloaded data off the connection associated with the response as below: 

ResponseData res = invoke(downloadUrl, contentType);
downloadedAssetStream = New ObjectInputStream(res.connection.getInputStream);

if(res.code != 200)
{
    //Stop hadoop job
}

For stopping the job, what do I need? I've heard somewhere that throwing an IOException would do, but I haven't been able to verify it yet.

Karthik. V
  • 33
  • 1
  • 3

1 Answers1

1

Job#killJob with the new MR API or RunningJob#killJob with the old MR API will kill the Job.

Praveen Sripati
  • 32,799
  • 16
  • 80
  • 117
  • 1
    So if I wish to kill the job from within the map() function, how do I refer to the job that I want to kill? In particular how do I get the job ID? – Karthik. V Dec 02 '11 at 09:56
  • To add to my earlier comment, my MyMapper class extends org.apache.hadoop.mapred.MapReduceBase and implements org.apache.hadoop.mapred.Mapper. In the (overriden) map() function, I would need to get the job instance to call the killJob(). If you could illustrate this, it'd be great. – Karthik. V Dec 02 '11 at 10:37
  • 1
    Not sure if killing the job from the map is a good practice, add the following code in the mapper to get the job id. `String jobId; public void configure(JobConf conf) { jobId = conf.get("mapred.job.id"); }` – Praveen Sripati Dec 02 '11 at 13:26
  • 1
    I suppose this question was inevitable... is it possible to retrieve the Job object from the JobID or Context objects from within the map function? – Karthik. V Dec 05 '11 at 06:20
  • 1
    http://stackoverflow.com/questions/8009802/is-there-a-way-to-access-number-of-successful-map-tasks-from-a-reduce-task-in-an/8054105#8054105 – Praveen Sripati Dec 05 '11 at 07:24
  • Does the Hadoop framework call configure() automatically? If not, how can configure() be provided with the JobConf object? I don't know where to access this object so as to pass it to the function. – Karthik. V Dec 06 '11 at 10:54
  • Yes - configure is automatically called by MR framework. Best way when not sure of something is to try it out. – Praveen Sripati Dec 06 '11 at 11:07