I felt doing my Mapper operation in Perl script but then I realized it would be easier to write Reducer in Python. Can Mapper and Reducer can work in different programming language?
Asked
Active
Viewed 555 times
1
-
3The obvious answer is *yes* – but I doubt this is your actual question. – amon Sep 15 '13 at 00:28
1 Answers
2
From the tags it sounds like you're using Amazon's Elastic Map Reduce service. This question is answered in their excellent FAQ:
Q: What programming languages does Amazon Elastic MapReduce support?
You can use Java to implement Hadoop custom jars. Alternatively, you may use other languages including Perl, Python, Ruby, C++, PHP, and R via Hadoop Streaming. Please refer to the Developer’s Guide for instructions on using Hadoop Streaming.
In other words, anything that can write to a file can be used in an EMR map or reduce step.
-
Exactly Amazon mentioned we could have many different language but didn't mention we can have combination of many, which seems obvious but just wanted to confirm as my job fails through EMR but working fine locally. May be have some different issue. Thanks anyways! – CtrlV Sep 15 '13 at 03:00