4

I'm very new with hadoop stream and have some difficulties with the partitioning.

According to what is found in a line, my mapper function either returns

key1, 0, somegeneralvalues # some kind of "header" line where linetype = 0

or

key1, 1, value1, value2, othervalues... # "data" line, different values, linetype =1

To properly reduce I need to group all lines having the same key1, and to sort them by value1, value2, and the linetype ( 0 or 1), something like:

1 0 foo bar...  # header first
1 1 888 999.... # data line, with lower value1
1 1 999 111.... # a few datalines may follow. Sort by value1,value2 should be performed
------------    #possible partition here, and only here in this example
2 0 baz foobar....   
2 1 123 888... 
2 1 123 999...
2 1 456 111...

Is there a way to ensure such partitioning ? so far I've tried to play with options such as

-partitioner,'org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner'
-D stream.num.map.output.key.fields=4 # please use 4 fields to sort data
-D mapred.text.key.partitioner.options=-k1,1 # please make partitions based on first key

or alternatively

-D num.key.fields.for.partition=1 # Seriously, please group by key1 !

which yet only brought rage and despair.

If it's worth mentioning it, my scripts work properly if I use cat data | mapper | sort | reduce and I'm using the amazon elastic map reduce ruby client, so I'm passing the options with 

--arg '-D','options' for the ruby script.

Any help would be highly appreciated ! Thanks in advance

aherve
  • 3,795
  • 6
  • 28
  • 41

2 Answers2

3

Thanks to ryanbwork I've been able to solve this problem. Yay !

The right idea was indeed to create a key that consists of a concatenation of the values. To go a little further, it is also possible to create a key that looks like

<'1.0.foo.bar', {'0','foo','bar'}>
<'1.1.888.999', {'1','888','999'}>

Options can then be passed to hadoop so that it can partition by the first "part" of the key. If I'm not mistaking in the interpretation, it looks like 

-partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartioner
-D stream.map.output.field.separator=. # I added some "." in the key
-D stream.num.map.output.key.fields=4  # 4 "sub-fields" are used to sort
-D num.key.fields.for.partition=1      # only one field is used to partition

This solution, based on what ryanbwork said, allows to create more reducers, while ensuring the data is properly splitted, and sorted.

aherve
  • 3,795
  • 6
  • 28
  • 41
2

After reading this post I'd propose modifying your mapper such that it returns pairs whose 'keys' include your key value, your linetype value, and the value1/value2 values all concatenated together. You'd keep the 'value' part of the pair the same. So for example, you'd return the following pairs to represent your first two examples:

<'10foobar',{'0','foo','bar'}>
<'11888999',{'1','888','999'}>

Now if you were to utilize a single reducer, all of your records would be get sent to the same reduce task and sorted in alphabetical order based on their 'key'. This would fulfill your requirement that pairs get sorted by key, then by linetype, then by value1 and finally value2, and you could access these values individually in the 'value' portion of the pair. I'm not very familiar with the different built in partioner/sort classes, but I'd assume you could just use the defaults and get this to work.

ryanbwork
  • 2,123
  • 12
  • 12