I've set up an Amazon EMR jobflow with 1 on-demand core node and 4 task nodes with bidding. When I run my task on only the core node each step finishes within 1 hour. When I'm lucky and have 1 core + 4 task nodes then steps usually finish within 10 minutes.
My problem is that when the task nodes are taken away by amazon, the rest of the task attempts start agonizing and they can agonize for 7-10 hours.
As you can see from the logs below, everything went OK from 08:56 (0%) - 09:01 (43%), but then task attempts started to fail. Now according to my calculations based on the fact that running the step on only the 1 core node would take less then an hour, I would expect the step to finish from 43%-100% in less than 1 hour. However it's agonizing for at least another 5+ hours: 09:01 - 14:30. This doesn't look to me normal (not to talk about the time and money wasted). How could I fix this? What can cause this?
2014-05-21 08:55:39,317 INFO com.amazon.ws.emr.hadoop.fs.EmrFileSystem (main): Opening 's3://test/log-parser/code/hadoop-script.sh' for reading
2014-05-21 08:55:45,555 INFO com.amazon.ws.emr.hadoop.fs.EmrFileSystem (main): Opening 's3://test/log-parser/code/hadoop-upload-script.sh' for reading
2014-05-21 08:55:52,990 INFO com.amazon.ws.emr.hadoop.fs.EmrFileSystem (main): Opening 's3://test/log-parser/code/log-parser.jar' for reading
2014-05-21 08:55:59,840 INFO com.innovid.logParser.LogParserMapReduce (main): LogParserMapReduce: waitingForCompletion
2014-05-21 08:56:00,190 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at /1.1.1.1:9022
2014-05-21 08:56:05,397 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input paths to process : 31
2014-05-21 08:56:05,426 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2014-05-21 08:56:05,434 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev c7d54fffe5a853c437ee23413ba71fc6af23c91d]
2014-05-21 08:56:05,669 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:135
2014-05-21 08:56:05,697 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): user.name is deprecated. Instead, use mapreduce.job.user.name
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.jar is deprecated. Instead, use mapreduce.job.jar
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2014-05-21 08:56:05,699 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
2014-05-21 08:56:05,699 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
2014-05-21 08:56:05,699 INFO org.apache.hadoop.conf.Configuration.deprecation (main): keep.failed.task.files is deprecated. Instead, use mapreduce.task.files.preserve.failedtasks
2014-05-21 08:56:05,699 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.job.name is deprecated. Instead, use mapreduce.job.name
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
2014-05-21 08:56:05,701 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
2014-05-21 08:56:05,701 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
2014-05-21 08:56:06,123 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1400094353_0186
2014-05-21 08:56:06,779 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1400094353_0186 to ResourceManager at /10.42.102.163:9022
2014-05-21 08:56:06,871 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://1.1.1.1:9046/proxy/application_1400094353_0186/
2014-05-21 08:56:06,872 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1400094353_0186
2014-05-21 08:56:15,441 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1400094353_0186 running in uber mode : false
2014-05-21 08:56:15,443 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2014-05-21 08:57:10,871 INFO org.apache.hadoop.mapreduce.Job (main): map 1% reduce 0%
2014-05-21 08:57:19,928 INFO org.apache.hadoop.mapreduce.Job (main): map 2% reduce 0%
2014-05-21 08:57:25,971 INFO org.apache.hadoop.mapreduce.Job (main): map 3% reduce 0%
2014-05-21 08:57:36,033 INFO org.apache.hadoop.mapreduce.Job (main): map 4% reduce 0%
2014-05-21 08:57:52,141 INFO org.apache.hadoop.mapreduce.Job (main): map 5% reduce 0%
2014-05-21 08:58:00,189 INFO org.apache.hadoop.mapreduce.Job (main): map 6% reduce 0%
2014-05-21 08:58:07,234 INFO org.apache.hadoop.mapreduce.Job (main): map 7% reduce 0%
2014-05-21 08:58:14,285 INFO org.apache.hadoop.mapreduce.Job (main): map 8% reduce 0%
2014-05-21 08:58:20,321 INFO org.apache.hadoop.mapreduce.Job (main): map 9% reduce 0%
2014-05-21 08:58:28,369 INFO org.apache.hadoop.mapreduce.Job (main): map 10% reduce 0%
2014-05-21 08:58:35,421 INFO org.apache.hadoop.mapreduce.Job (main): map 11% reduce 0%
2014-05-21 08:58:44,481 INFO org.apache.hadoop.mapreduce.Job (main): map 12% reduce 0%
2014-05-21 08:58:51,530 INFO org.apache.hadoop.mapreduce.Job (main): map 13% reduce 0%
2014-05-21 08:58:59,583 INFO org.apache.hadoop.mapreduce.Job (main): map 14% reduce 0%
2014-05-21 08:59:06,625 INFO org.apache.hadoop.mapreduce.Job (main): map 15% reduce 0%
2014-05-21 08:59:12,697 INFO org.apache.hadoop.mapreduce.Job (main): map 16% reduce 0%
2014-05-21 08:59:20,766 INFO org.apache.hadoop.mapreduce.Job (main): map 17% reduce 0%
2014-05-21 08:59:26,804 INFO org.apache.hadoop.mapreduce.Job (main): map 18% reduce 0%
2014-05-21 08:59:33,868 INFO org.apache.hadoop.mapreduce.Job (main): map 19% reduce 0%
2014-05-21 08:59:39,907 INFO org.apache.hadoop.mapreduce.Job (main): map 20% reduce 0%
2014-05-21 08:59:46,959 INFO org.apache.hadoop.mapreduce.Job (main): map 21% reduce 0%
2014-05-21 08:59:54,003 INFO org.apache.hadoop.mapreduce.Job (main): map 22% reduce 0%
2014-05-21 09:00:01,052 INFO org.apache.hadoop.mapreduce.Job (main): map 23% reduce 0%
2014-05-21 09:00:08,108 INFO org.apache.hadoop.mapreduce.Job (main): map 24% reduce 0%
2014-05-21 09:00:16,196 INFO org.apache.hadoop.mapreduce.Job (main): map 25% reduce 0%
2014-05-21 09:00:22,241 INFO org.apache.hadoop.mapreduce.Job (main): map 26% reduce 0%
2014-05-21 09:00:29,288 INFO org.apache.hadoop.mapreduce.Job (main): map 27% reduce 0%
2014-05-21 09:00:36,328 INFO org.apache.hadoop.mapreduce.Job (main): map 28% reduce 0%
2014-05-21 09:00:43,410 INFO org.apache.hadoop.mapreduce.Job (main): map 29% reduce 0%
2014-05-21 09:01:22,288 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2014-05-21 09:01:26,318 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 0%
2014-05-21 09:01:29,334 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 0%
2014-05-21 09:01:32,351 INFO org.apache.hadoop.mapreduce.Job (main): map 33% reduce 0%
2014-05-21 09:01:35,368 INFO org.apache.hadoop.mapreduce.Job (main): map 34% reduce 0%
2014-05-21 09:01:38,384 INFO org.apache.hadoop.mapreduce.Job (main): map 35% reduce 0%
2014-05-21 09:01:40,395 INFO org.apache.hadoop.mapreduce.Job (main): map 36% reduce 0%
2014-05-21 09:01:41,401 INFO org.apache.hadoop.mapreduce.Job (main): map 37% reduce 0%
2014-05-21 09:01:43,416 INFO org.apache.hadoop.mapreduce.Job (main): map 38% reduce 0%
2014-05-21 09:01:45,428 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 0%
2014-05-21 09:01:47,439 INFO org.apache.hadoop.mapreduce.Job (main): map 40% reduce 0%
2014-05-21 09:01:49,451 INFO org.apache.hadoop.mapreduce.Job (main): map 41% reduce 0%
2014-05-21 09:01:51,474 INFO org.apache.hadoop.mapreduce.Job (main): map 42% reduce 0%
2014-05-21 09:01:54,491 INFO org.apache.hadoop.mapreduce.Job (main): map 43% reduce 0%
2014-05-21 09:06:44,031 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 0%
2014-05-21 09:07:44,353 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 09:22:04,475 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000106_0, Status : FAILED
2014-05-21 09:22:05,499 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 09:32:10,244 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 09:37:25,611 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000079_0, Status : FAILED
2014-05-21 09:37:25,613 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000094_0, Status : FAILED
2014-05-21 09:37:25,614 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000107_0, Status : FAILED
2014-05-21 09:37:25,615 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000088_0, Status : FAILED
2014-05-21 09:37:26,620 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 09:48:50,521 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 09:52:45,491 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000081_0, Status : FAILED
2014-05-21 09:52:46,495 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 10:05:30,717 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 10:08:06,373 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000090_0, Status : FAILED
2014-05-21 10:08:07,377 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 10:18:50,064 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 10:23:28,146 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000110_0, Status : FAILED
2014-05-21 10:23:29,150 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 10:35:29,790 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 10:38:48,507 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000104_0, Status : FAILED
2014-05-21 10:38:49,511 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 10:52:10,333 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 10:54:09,754 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000108_0, Status : FAILED
2014-05-21 10:54:10,758 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 11:05:30,115 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 11:09:29,958 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000095_0, Status : FAILED
2014-05-21 11:09:30,961 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 11:22:10,617 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 11:24:51,159 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000084_0, Status : FAILED
2014-05-21 11:24:51,160 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000116_0, Status : FAILED
2014-05-21 11:24:52,163 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 11:35:29,377 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 11:40:12,354 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000078_0, Status : FAILED
2014-05-21 11:40:13,358 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 11:52:09,812 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 11:55:32,516 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000113_0, Status : FAILED
2014-05-21 11:55:33,520 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 12:08:50,313 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 12:10:53,733 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000093_0, Status : FAILED
2014-05-21 12:10:54,739 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 12:22:10,023 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 12:26:14,825 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000097_0, Status : FAILED
2014-05-21 12:26:15,828 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 12:38:50,340 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 12:41:35,872 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000114_0, Status : FAILED
2014-05-21 12:41:36,876 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 12:41:45,906 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000049_0, Status : FAILED
2014-05-21 12:41:55,938 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000053_0, Status : FAILED
2014-05-21 12:41:56,942 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 12:42:05,974 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000041_0, Status : FAILED
2014-05-21 12:42:06,977 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 12:42:16,009 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000062_0, Status : FAILED
2014-05-21 12:42:17,013 INFO org.apache.hadoop.mapreduce.Job (main): map 96% reduce 0%
2014-05-21 12:42:26,043 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000045_0, Status : FAILED
2014-05-21 12:42:36,077 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000050_0, Status : FAILED
2014-05-21 12:42:37,080 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 0%
2014-05-21 12:52:09,942 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 12:55:29,568 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 12:56:56,841 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000109_0, Status : FAILED
2014-05-21 12:56:56,842 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000099_0, Status : FAILED
2014-05-21 12:56:57,845 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 12:57:06,876 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000044_0, Status : FAILED
2014-05-21 12:57:07,879 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 12:57:16,909 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000072_0, Status : FAILED
2014-05-21 12:57:17,912 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 12:57:25,974 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000057_0, Status : FAILED
2014-05-21 12:57:26,977 INFO org.apache.hadoop.mapreduce.Job (main): map 96% reduce 0%
2014-05-21 12:57:36,006 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000056_0, Status : FAILED
2014-05-21 12:57:46,038 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000069_0, Status : FAILED
2014-05-21 12:57:47,041 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 0%
2014-05-21 12:57:56,069 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000076_0, Status : FAILED
2014-05-21 12:57:57,072 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 0%
2014-05-21 12:58:06,102 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000042_0, Status : FAILED
2014-05-21 12:58:07,105 INFO org.apache.hadoop.mapreduce.Job (main): map 93% reduce 0%
2014-05-21 12:58:16,134 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000071_0, Status : FAILED
2014-05-21 12:58:26,166 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000051_0, Status : FAILED
2014-05-21 12:58:27,171 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 0%
2014-05-21 12:58:36,207 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000039_0, Status : FAILED
2014-05-21 12:58:37,210 INFO org.apache.hadoop.mapreduce.Job (main): map 91% reduce 0%
2014-05-21 12:58:46,240 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000060_0, Status : FAILED
2014-05-21 12:58:47,243 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 0%
2014-05-21 12:58:56,271 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000061_0, Status : FAILED
2014-05-21 13:08:50,122 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 13:12:09,752 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 13:12:17,779 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000091_0, Status : FAILED
2014-05-21 13:12:17,780 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000083_0, Status : FAILED
2014-05-21 13:12:17,781 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000115_0, Status : FAILED
2014-05-21 13:12:18,784 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 13:12:27,813 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000070_0, Status : FAILED
2014-05-21 13:12:28,816 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 13:25:30,288 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 13:27:37,695 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000085_0, Status : FAILED
2014-05-21 13:27:38,698 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 13:27:47,729 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000058_0, Status : FAILED
2014-05-21 13:27:57,762 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000063_0, Status : FAILED
2014-05-21 13:27:58,765 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 13:38:50,829 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 13:42:58,603 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000086_0, Status : FAILED
2014-05-21 13:42:59,606 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 13:43:08,636 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000077_0, Status : FAILED
2014-05-21 13:55:30,006 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 13:58:19,532 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000098_0, Status : FAILED
2014-05-21 13:58:20,535 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 14:08:49,521 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 14:13:40,429 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000101_0, Status : FAILED
2014-05-21 14:13:41,433 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 14:13:50,462 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000059_0, Status : FAILED
2014-05-21 14:25:29,656 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 14:29:01,327 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000103_0, Status : FAILED
2014-05-21 14:29:02,330 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 14:29:11,361 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000067_0, Status : FAILED
2014-05-21 14:29:21,394 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000075_0, Status : FAILED
2014-05-21 14:29:22,397 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 14:29:31,427 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000046_0, Status : FAILED
2014-05-21 14:29:32,430 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 14:29:41,458 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000047_0, Status : FAILED
2014-05-21 14:29:42,461 INFO org.apache.hadoop.mapreduce.Job (main): map 96% reduce 0%
2014-05-21 14:29:51,491 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000054_0, Status : FAILED
2014-05-21 14:30:01,550 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000068_0, Status : FAILED
2014-05-21 14:30:02,554 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 0%
2014-05-21 14:30:11,591 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000048_0, Status : FAILED
2014-05-21 14:30:12,594 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 0%
2014-05-21 14:30:21,626 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000038_0, Status : FAILED
2014-05-21 14:30:22,632 INFO org.apache.hadoop.mapreduce.Job (main): map 93% reduce 0%
2014-05-21 14:30:31,660 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000052_0, Status : FAILED
2014-05-21 14:30:41,691 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000043_0, Status : FAILED
2014-05-21 14:30:42,694 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 0%
2014-05-21 14:30:51,724 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000065_0, Status : FAILED
2014-05-21 14:30:52,727 INFO org.apache.hadoop.mapreduce.Job (main): map 91% reduce 0%
2014-05-21 14:42:09,877 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 14:44:22,301 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000082_0, Status : FAILED
2014-05-21 14:44:22,302 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000080_0, Status : FAILED
2014-05-21 14:44:22,303 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000102_0, Status : FAILED
2014-05-21 14:44:22,304 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000096_0, Status : FAILED
2014-05-21 14:44:23,307 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 14:44:32,338 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000055_0, Status : FAILED
2014-05-21 14:44:33,341 INFO org.apache.hadoop.mapreduce.Job (main): map 96% reduce 0%
2014-05-21 14:44:42,371 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000073_0, Status : FAILED
2014-05-21 14:44:52,405 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000074_0, Status : FAILED
2014-05-21 14:44:53,408 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 0%
2014-05-21 14:45:02,441 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000064_0, Status : FAILED
2014-05-21 14:45:03,446 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 0%
2014-05-21 14:45:12,480 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000066_0, Status : FAILED
2014-05-21 14:45:13,486 INFO org.apache.hadoop.mapreduce.Job (main): map 87% reduce 0%
2014-05-21 14:45:22,520 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000040_0, Status : FAILED
