Cascading(buffer) implementation

Question

I need to create a buffer in cascading hadoop.

Suppose i have fields :

member_id,amountpaid,diadnosis_id,diagnosis_description,superGrouper_id,superGrouper_descriptiion,grouperId,grouperDescription

I need to

group the fields from member_id and superGrouper_id
send these information using every pipe to a buffer
the buffer output should be :member_id, highest paid sorting from superGrouper, highest paid sorting from grouperId, highest paid from diagnosis_id, along with their description...

Please help me creating a buffer. Thanks in advance

Could you post the input and output you expect. Then, I would try to provide you answer. Actually its not quite clear. — pramesh, May 05 '17 at 06:15

score 1 · Answer 1 · edited May 23 '15 at 11:56

1

You don't need a custom buffer. Use the built in Max aggregator from Cascading. Cascading Docs

You then just need to run the Max after a GroupBy.

pipe = new GroupBy(pipe, new Fields("member_id", "superGrouper_id");
pipe = new Every(pipe, new Fields("amountpaid"), new Max(new Fields("max_paid"));

edited May 23 '15 at 11:56

victorkt

13,992
9
52
51

answered Apr 04 '13 at 19:31

Brian Ethier

169
3

score 0 · Answer 2 · edited May 23 '15 at 11:56

0

you can do the following:

pipe = new GroupBy(pipe, new Fields("member_id", "superGrouper_id"), new Fields("superGrouper", "grouperId", "")); 
pipe = new Every(pipe, FirstNBuffer(int n));

I am sorry if I am wrong. Your question is not quite clear.

edited May 23 '15 at 11:56

victorkt

13,992
9
52
51

answered May 21 '15 at 11:21

pramesh

1,914
1
19
30

Cascading(buffer) implementation

2 Answers2