How to prevent a Hadoop job to fail when directory is empty?

Question

I have a job that fails when there is no files in the input directory. The exception i get is the following:

org.apache.hadoop.mapreduce.lib.input.InvalidInputException:Input Pattern maprfs:/profile/*

I know this exception is coming from the FileInputFormat class. What would be the best approach for this problem. Would overwriting the InputFormat Class be the only solution?

Thank you

what is the requirement to do so? – YoungHobbit Sep 22 '15 at 16:20 — YoungHobbit, Sep 22 '15 at 16:20

score 1 · Accepted Answer · answered Sep 22 '15 at 17:06

1

Put an empty file in your input path.
At the beginning of your driver list your Input path(S). If there isn’t a file, then exit without submit the job.

answered Sep 22 '15 at 17:06

RojoSam

1,476
12
15

Thank you. This is exactly what I did. – danilo Sep 22 '15 at 19:25

How to prevent a Hadoop job to fail when directory is empty?

1 Answers1