Hadoop MapReduce: Two values as key in Mapper-Reducer

Question

how can I build a key with two components? The reason for this is I have an undirected graph. There is an edge between two nodes A and B if A and B were associated through a communication (the direction is irrelevant). This communication has a numerical parameter. So what I would like to achieve is to have a key which combines A and B together as a set, so that the communication from A to B and B to A can be considered equivalent and be summed up to get stats

Say:

A B 5

B A 10

The key then should be semantically "A or B together", so that the set containing A and B as key should have the value 5+10=15.

The wordcount example has as key the specific words. In my case, I want to have as key a set with two components. During the map and reduce phases, I would like to sum as long as A to B or B to A satisfies.

Thx!

score 2 · Answer 1 · edited Sep 14 '20 at 05:02

2

You need custom key, with own comparison rules. You doing it by implementing WritableComparable over you class containing graph link information. This is example / explanations: https://web.archive.org/web/20130729080313/http://developer.yahoo.com/hadoop/tutorial/module5.html

edited Sep 14 '20 at 05:02

Nick ODell

15,465
3
32
66

answered Jun 05 '12 at 08:32

David Gruzman

7,900
1
28
30

Thanks, as always amazed how fast you guys respond! – Bob Jun 05 '12 at 10:12

score 0 · Accepted Answer · answered Jun 06 '12 at 14:48

0

In addition to the (correct) answer by David: If your problem has to with graphs then have a look at http://incubator.apache.org/giraph/ also.

answered Jun 06 '12 at 14:48

Niels Basjes

10,424
9
50
66

Thanks for the reference, it is definitely worth to give a look. Do you know other tools which can visualize big graphs? Say I have nodes and edges information in some format. I mean tools similar to Gephi? Thx. – Bob Jun 08 '12 at 11:42

Hadoop MapReduce: Two values as key in Mapper-Reducer

2 Answers2

Linked