My MapReduce job uses MultipleOutputs to write files into three separate directories.
My reducer count is 400, which is idle for files written into two directories. For the 3rd directory, I am trying to reduce the number of counter files written since the file size is tiny. So 400 small counter files consume many blocks in HDFS.(I don't want that to happen)
I want to keep the reduce count the same and only reduce the files written into one directory. Does MapReduce support something like a spark coalesce? Or can multipleoutputs help in someway to just write into 1 or 2 files instead of 400 ?