Creating CombineFileSplit in Java of Hadoop?

Question

CombineFileSplit has a constructor:

CombineFileSplit(Path[] files, long[] start, long[] lengths, String[] locations) 

is it necessary for the locations array to be the same size as files array, because the file can reside in many locations/nodes?

Answer 1

It is not necessary that the locations array to be the same size as files array, because from the FileSplit's constructor: public FileSplit(Path file, long start, long length, String[] hosts) you can conclude that a file can have many locations/hosts. Also if you see the source code of the constructor of CombineFileSplit, you can pass an array of empty strings. I think the locations array is meant to be used by you so if you know the locations of the file that's it you can provide it or pass empty strings.