2

I know that during the intermediate steps between mapper and reducer, hadoop will sort and partition the data on its way to the reducer.

Since I am dealing with already partitioned data in my input to the mapper, is there a way to take advantage of it and possibly accelerate the intermediate processing so no more sorting or grouping-by will take place?

Adding some details:

As I store data on S3, let's say I only have two files in my bucket. First file will store records of the lower half users ids, the other file will store values of the upper half of user ids. Data in each file is not necessarily sorted, but it is guaranteed that all data pertaining to a user is located in the same file.

Such as:

\mybucket\file1
\mybucket\file2

File1 content:
User1,ValueX
User3,ValueY
User1,ValueZ
User1,ValueAZ

File2 content:
User9,ValueD
User7,ValueB
User7,ValueD
User8,ValueB

From what I read, I can use a streaming job and two mappers and each mapper will suck in one of the two files, but the whole file. Is this true?

Next, Let's say the mapper will only output a unique Key just once, with the associated value being the number of occurrences of that Key. (which I realize it is more of a reducer responsibility, but just for our example here)

Can the sorting and partitioning of those output keys from the Mapper be disabled and let them fly freely to the reducer(s) ?

Or to give another example: Imagine all my input data contains just one line for each Unique Key, and I don't need that data to be sorted in the final output of the reducer. I just want to Hash the Value for each Key. Can I disable that sorting and partitioning step before the reducer?

Gabriel Burete
  • 140
  • 6
  • Could you shed more light on what you mean by "partitioned input to mapper", maybe a sample of input data? – Chaos Jun 25 '13 at 21:21

1 Answers1

0

Although for the files shown above you'll get 2 mappers, it can't be guaranteed always. Number of mappers depend upon the number of InputSplits created from the input data. If your files are big you might have more than one mappers.

Partitioning is merely a way to tell which key/value goes to which reducer. If you disable it then you either need some other way to do this or you'll end up with performance degradation, as the inputs to reducers will be uneven. A particular reducer might get all of the input or a particular reducer might get zero input. I can't see any performance gain here. Of course, if you think your custom partitioner fits better into the situation you could definitely do that. But skipping partitioning doesn't sound logical to me. The default partitioning behavior depends on hash itself. After a mapper emits its output keys are hashed to find out which set of key/value pairs goes to which reducer.

And if your data is already sorted and you want to skip the sorting phase in your MR job, you might find the patch provided in response to this JIRA useful. Issue is not closed yet, but it would definitely help you in getting started.

HTH

Tariq
  • 34,076
  • 8
  • 57
  • 79
  • Thanks for that JIRA link. So would you say that there is no real advantage on keeping my data grouped by user id in my input files and not having a certain group span multiple files? I might be coming from the mindset where affinity of data to a certain worker speeds things up, and I am trying to configure MR like that: I want a mapper to process a whole file, and I want the Reducer to be very simple and just aggregate results together. – Gabriel Burete Jun 25 '13 at 22:57
  • If you intend to process a single file by just 1 mapper set "isSplittable" to "true" in your InputFormat. This'll ensure that your file doesn't get split, thus going to a single mapper as a whole. When you feed input to MR job, it doesn't matter how the data is arranged in a file. I don't see any advantage at the mapper level. But you can gain some speedup during shuffle, sort and merge phase though. – Tariq Jun 26 '13 at 01:54