EMR No output for a long time

Asked Jan 18 '13 at 11:42

Active Jan 18 '13 at 11:42

Viewed 110 times

I have a MapReduce job written in python using MRJob library. The job takes around 30 mins to complete on my local machine. While running the same job on the EMR, I am seeing no output for a long time (~=1hr). I had to close down the job. Also the jobs which take very small time on my local machine run very well on EMR. I tried increasing the timeout but the task tracker kills the reducer after the timeout. I get the following error message

`Task attempt_201301181027_0001_r_000000_0 failed to report status for 3600 seconds.Killing!

The EMR creates 4 mappers and one reducer for my job. Also the reducer code that I have written requires a lot of looping through the entire data set which is around 11-12 MB. I Expect the job to finish in around same time as that of my local machine but it doesnt happen. Whats the solution??`

asked Jan 18 '13 at 11:42

Read Q

1,405
2
14
26

Can you share your reducer code? – Charles Menguy Jan 18 '13 at 19:07
Its a long code actually! The strange thing is, on my jobtracker I see both map-100% and reduce=100%. – Read Q Jan 19 '13 at 05:37
It's possible that you just have a lot of data to upload to S3 - maybe copy your code to pastebin and share the link. – Charles Menguy Jan 19 '13 at 05:42
Input data is around 12 MB. I am taking the output on my local machine – Read Q Jan 19 '13 at 05:49
Can't really help without any idea of how you setup your job and what the reducer looks like – Charles Menguy Jan 19 '13 at 05:52
http://pastebin.com/M46RA23u – Read Q Jan 19 '13 at 05:53

EMR No output for a long time

0 Answers0