0

I want to run MapReduce application using Hadoop 2.6.5 (in my own native cluster) and I want to update some things in YARN thus I have seen that I can write my own YARN application (https://hadoop.apache.org/docs/r2.6.5/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html). But it seems like if I am running YARN in this way (with YARN client) then, I can't use the MapReduce paradigm (Map and Reduce Function) with a Job class.

Is there any option to write my own ApplicationManager and using the MapReduce paradigm with the simple Job class?

These are some useful examples I have found regarding writing YARN application https://github.com/noitcudni/yarn https://github.com/hortonworks/simple-yarn-app https://github.com/blrunner/yarn-beginners-examples/blob/master/src/main/java/com/wikibooks/hadoop/yarn/examples/MyClient.java

*Using Spring or Twill will result in the same problem.

Or Raz
  • 39
  • 2
  • 11
  • What exactly do you want to change in YARN? – Ben Watson Oct 04 '18 at 10:57
  • There's a difference between processing data and just running a disturbed microservice-like application that just sits in YARN... Nothing is preventing your own application from separately executing MapReduce, but not all YARN applications are built on that model – OneCricketeer Oct 04 '18 at 13:58
  • I want to change the allocation of tasks (placement or partitioning). @cricket_007 Do you know how to combine these two? I am not sure what to write a MapReduce Job that uses my newAM or YARN application with AM that will satisfy the procedure of allocating Map and Reduce tasks with their classes and their other classes such as Partitioner. – Or Raz Oct 07 '18 at 07:21
  • YARN already has node labels to change how jobs are placed. What else do you need to change? – OneCricketeer Oct 08 '18 at 00:41
  • Is it change how the way they placed? As far as I concern it is not optimal, and there is a room for improvement (which I want to introduce). I want to update the task allocation with a hint of a better allocation to the RM and AM. Do you know how to make my idea? Or at least the combination of YARN and MapReduce apps from above? (BTW it was introduced in Apache Hadoop 2.7.3, and I am using 2.6.5) – Or Raz Oct 08 '18 at 04:10

0 Answers0