0

I have a very specific problem in Hadoop.

I have two files userlist and *raw_data*. Now raw_data is a pretty big file and userlist is a comparatively smaller than the other file.

I have to first identify the number of mappers and my userlist has to broken down to pieces equal to the number of mappers. Later it has to be loaded into distributed cache and it has to compare with userlist and perform some analytics and write it to reducer.

Please suggest.

thank you.

Thirumalreddy_Bandi
  • 626
  • 6
  • 8
  • Did you accidentaly write `"...it has to compare with userlist and perform some analytics"`, instead of `"...it has to compare with raw-data and perform some analytics"`? – vefthym Feb 15 '14 at 18:12

1 Answers1

0

I do not understand why you want to partition the userlist file. If it is small then, load the entire userlist file to distributed cache. Then in the setup method of map class, every mapper will have access to the entire userlist file. Moreover, you can find out the number of mappers and partition it as you prefer in the setup method.

justin waugh
  • 885
  • 3
  • 12
  • 22