0

I am getting this error Failed to report status for 606 seconds in Mapper phase.I don't have a reducer phase.I referred to [https://stackoverflow.com/questions/16056707/failed-to-report-status-for-600-seconds-killing-reporting-progress-in-hadoop?rq=1],[https://stackoverflow.com/questions/9109177/hadoop-job-task-fail-to-report-status-for-601-seconds?rq=1] and few others.After going thru the solutions provided in these questions,I have added the counter enum in my program and also reporting the progress.However still getting the error.I am new to hadoop and want to make sure wht I am doing is correct or not.

    public class Map extends Mapper<LongWritable,Text,Text,Text> {

        static enum Counter { SOME_COUNTER }

     public void map(LongWritable key, Text value, Context context) throws 
                    IOException, InterruptedException {


            //increment counter and report progress
                context.getCounter(ScanCounter.SCAN_COUNTER).increment(1);
                context.progress();
    //my map activity
    //Read a file and do some operation on it.
        String[] cmd = {
                            "/bin/sh",
                            "-c",
                            "hadoop dfs -cat " + path + "| someoperation"
                            };
    try {

                    Process p = Runtime.getRuntime().exec(cmd);
                    BufferedReader reader = 
                                    new BufferedReader(new InputStreamReader(p.getInputStream()));
                    context.progress();

                  //read the output here as value

                        context.write(new Text(path), new Text(value));
    }
}
}

Thanks!

Community
  • 1
  • 1
user3273354
  • 23
  • 1
  • 1
  • 6
  • Check for swapping or network issues (to the master) on your slave. Besides that, are you doing anything that takes long? From the code you've given we have no clue what this might be. – Thomas Jungblut Jul 10 '15 at 15:22
  • Hi Thomas,Yes I am scanning a file and doing some operation on it.The size of one of the file is large which causes the issue..I will update that to my question.But the way I am reporting progress,is it correct ? Or do we have to add that line inside the loop which does operation on my file? – user3273354 Jul 10 '15 at 15:52
  • `do we have to add that line inside the loop which does operation on my file` if it takes longer than 10 minutes to process then yes it should go into your loop that does the work. – Thomas Jungblut Jul 10 '15 at 16:20
  • Hey Thomas,how to report status for hadoop fs -cat /path/to/largefile ?This is the one causing problem for me. – user3273354 Jul 10 '15 at 19:47
  • can you please attach the WHOLE code and not just a comment? – Thomas Jungblut Jul 10 '15 at 19:58
  • do you need `| someoperation` or can you use Java code for that? I don't know why you are piping this through bash. – Thomas Jungblut Jul 10 '15 at 20:08
  • yes I need that..but if I am not successful in reporting the progress I wil change the logic..first I need to try to report the progress. – user3273354 Jul 10 '15 at 20:11
  • I changed the | operation to java code and was able to report the progress.Thanks! – user3273354 Jul 19 '15 at 07:31

0 Answers0