Multiple outputs suddenly not writing any output?

Question

Suddenly multiple outputs not writing any output to the destination.

I use a custom implementation of multiple outputs, where, I just changed:

  if((ch == '/') || (ch == ':')||(ch == '-')||(ch =='.'))
      {
          continue;
      }

in the method, as shown below. But the same was working all along, and suddenly it is not working. It is not writing anything to the output directory:

/home/users/mlakshm/

pls help!!!

 private static void checkTokenName(String namedOutput) {
    if (namedOutput == null || namedOutput.length() == 0) {
      throw new IllegalArgumentException(
        "Name cannot be NULL or emtpy");
    }
    for (char ch : namedOutput.toCharArray()) {
      if ((ch >= 'A') && (ch <= 'Z')) {
        continue;
      }
      if ((ch >= 'a') && (ch <= 'z')) {
        continue;
      }
      if ((ch >= '0') && (ch <= '9')) {
        continue;
      }
      if((ch == '/') || (ch == ':')||(ch == '-')||(ch =='.'))
      {
          continue;
      }
      throw new IllegalArgumentException(
        "Name cannot be have a '" + ch + "' char");
    }
  }

I am not getting any exception. But nothing is being written — Mahalakshmi Lakshminarayanan, Mar 17 '13 at 14:53
@MahalakshmiLakshminarayanan But you're not writing anything.. — Maroun, Mar 17 '13 at 15:02
@Jeffrey, Sachin I am just writing mos.getCollector("/home/users/mlakshm/RemFile/rfile, reporter).collect(new Text(str_key), new Text(key_val)); The str_key, and str_val, I even gave some string like: "hi" in that place, but no use. Output folder is getting created, but nothing is being written. I am wondering if this is some issue related to not enough space being available in dfs. — Mahalakshmi Lakshminarayanan, Mar 17 '13 at 15:09

score 0 · Answer 1 · answered Mar 18 '13 at 20:43

As you may notice the method checkTokenName() is to make sure that the output name is valid. Now, you are trying to modify the very integral part of MultipleOutputs which you shouldn't. There are a number of reasons that characters like /,:,. and - aren't allowed in the first place:

Many file systems doesn't allow some of these characters in a filename
Using MultipleOutputs, one may write to multiple files but in a designated directory, not at any desired location.
As you may have noticed, for a following named output:

// Defines additional single text based output 'text' for the job MultipleOutputs.addNamedOutput(job, "text", TextOutputFormat.class, LongWritable.class, Text.class); The output file names would be text-0000, text0001 etc.

So if you want to write in different directories, then better go for overriding MultipleTextOutputFormat; and if you aren't in liberty of using old API, then you better write to HDFS/S3 yourself rather than relying on hadoop to do that.

Thanks for the answer.. The guidelines were useful. The multipleoutputs was working fine. I made a mistake in the configure method, which reads the files written using multiple outputs via distributed cache. I used configure(conf, reprter) instead of configure(conf), so the program never entered configure method, and no files were read and i mistook it as no files were written by multiple outputs. — Mahalakshmi Lakshminarayanan, Mar 20 '13 at 19:43

Multiple outputs suddenly not writing any output?

1 Answers1