0

I started with hadoop recently, and I’m trying to use it with giraph (because i need it for manipulating huge graphs). So, for building giraph (1.1.0, latest stable version), I chose the 2.4.0 release of hadoop (according to this link, it's the latest version supported in Amazon EMR). I'm still doing local test, and I'm not even close to a full deploy in AWS, but I'm trying to do my stuff working towards that objective (programming in Amazon EMR).

But, when I was building giraph with Maven (I'm avoiding the use of YARN profile), I hit the problem detected in this bug (HADOOP-10547). So, I don't know which is the best option to choose from here, I hope that someone can help me here:

  • Choose the 2.2 version and try to build giraph with it (it's the most advanced version, previous to 2.4, available). The downside of this option, is that i'm just starting and i want to use the latest hadoop available version.
  • Download the source of hadoop 2.4, fix HADOOP-10547 by myself (because it appears that the 2.4 version of hadoop for AWS doesn't have this fix), build it, and try to rebuild giraph after. I only be building hadoop by myself, in order of fixing that bug, I don't have plans for keep modifying it after that. The downside of this option, it's that I'm not sure if it's possible do that again in an AWS deploy (maybe with bootstraps actions?)

Maybe there is another option that I'm not considering in order of use hadoop 2.4.0 and giraph 1.1.0 in Amazon EMR.

chomp
  • 1,352
  • 13
  • 31

1 Answers1

0

Some time ago i completed this form, when i saw that HADOOP-10547 wasn't fixed in the 2.4 version of hadoop for AWS, telling them that would very nice if they can fix that bug.

A few days ago a member of the Amazon AWS team send me this email:

I’ll forward the feedback wrt to HADOOP-10547 patch to the team that integrates the Apache releases into EMR. It’s always important to get this type of feedback from our customers so we can provide what they want, so thanks for reaching out.

So maybe in some time we will have a definitive solution for this.

chomp
  • 1,352
  • 13
  • 31