0

Hadoop(and Java) neophyte here. I needed some help with using MultipleTextOutputFormat to control the output filename in MapReduce.

Currently I am using it this way. And it seems to work fine. However what I am trying to change is the usage of the fields that get picked to determine the filename.

Instead of hardcoding them to field[0] or field[3](as is the case in the sample), I would like to pick this up(in some dynamic fashion) from say, JobConf as field[jobConf.get("id.offset")] or field[jobConf[get("date.offset")]. Does anyone here know how I could go about doing this (or something to this effect i.e. it doesn't have to be JobConf per se)?

Any pointers/suggestions/tips et al. would be most appreciated. Thanks.

decimus phostle
  • 1,040
  • 2
  • 13
  • 28

2 Answers2

4

It depends on if your custom parameters differ based on a job or on key/value pairs.

You can get the JobConf object if you override the getRecordWriter() method. This is the method that calls generateFileNameForKeyValue() (check out the implementation in the class MultipleOutputFormat, in Hadoop source). You could just call super and then do whatever you want with the parameter you pass in the JobConf object.

If your parameters differ for different key/value pairs, you can send the parameter as part of the key or the value. And then override getActualKey() or getActualValue() in your MultipleTextOutputFormat class to get the actual key or value object that you want.

Hope this helps.

Satyajit Chetri
  • 76
  • 1
  • 3
0

You could add settings to the conf object.

Configuration conf = new Configuration();
conf.set("id.offset", 0);
conf.set("date.offset", 3);
...   
JobConf jobConf = new JobConf(conf, MyJob.class);

In your fileName function, you could then use it as you are looking to; field[jobConf.get("id.offset")] and field[jobConf[get("date.offset")].

One small note here: I haven't used JobConf, so I'm don't actually know how to pull the settings out. I use Job, and the map function has the context object which I use as context.getConfiguration() to get the Configuration object, conf, and get the data as int idOffset = conf.get("id.offset");.

hth

QuinnG
  • 6,346
  • 2
  • 39
  • 47
  • Nija: I am already doing something along those lines in the Mapper. I guess another way to phrase my question would be - in an instance (of a subclass) of `MultipleTextOutputFormat`, how do I get a handle to a `JobConf` object? – decimus phostle Aug 09 '11 at 14:13