Hadoop using c++ pipes: How to call Mapper.cleanup()

Question

Using C++ pipes api(1.2.0), How can I possibly get a call in Mapper.cleanup() after the map() phase of the mapper? Basically for each chunk I want to store my records in-memory during the map phase and then apply some processing afterwards.

Any hints are welcome, Thanks,

How about using destructor? You can save the context and use it in the destructor. — zsxwing, May 30 '13 at 15:07
Not too familiar with Pipes, but can't you just wait for std input to close - and at that stage all the records have been passed. Then you can run your cleanup and output the records to stdout? — Chris White, May 30 '13 at 15:37
@zsxwing - yes, that possibly could work. I'll give it a try. Thanks, — srbhkmr, May 31 '13 at 15:02
@ChrisWhite - The thing is I am currently relying on Mapper.map() calls to read in the data, so I have no way of knowing which call to map() is going to be the last call. There has to be better ways of doing this. I'm currently in the Exploration phase :) — srbhkmr, May 31 '13 at 15:04

score 0 · Answer 1 · answered May 31 '13 at 17:52

The Mapper c++ class extends Closable:

class Mapper: public Closable {
public:
  virtual void map(MapContext& context) = 0;
};

and Closable has the following signature:

class Closable {
public:
  virtual void close() {}
  virtual ~Closable() {}
};

So (not being a c++ programmer), i'm guessing you just need to write your logic in a method named close

Hadoop using c++ pipes: How to call Mapper.cleanup()

1 Answers1