Hadoop DistCp handle same file name by renaming

Question

Is there any way to run DistCp, but with an option to rename on file name collisions? Maybe it's easiest to explain with an example.

Let's say I'm copying to hdfs:///foo to hdfs:///bar, and foo contains these files:

hdfs:///foo/a
hdfs:///foo/b
hdfs:///foo/c

and bar contains these:

hdfs:///bar/a
hdfs:///bar/b

Then after the copy, I'd like bar to contain something like:

hdfs:///bar/a
hdfs:///bar/a-copy1
hdfs:///bar/b
hdfs:///bar/b-copy1
hdfs:///bar/c

If there is no such option, what might be the most reliable/efficient way to do this? My own home-grown version of distcp could certainly get it done, but that seems like it could be a lot of work and pretty error-prone. Basically, I don't care at all about the file names, just their directory, and I want to periodically copy large amounts of data into a "consolidation" directory.

I know; I don't want to overwrite the old files. I want both the old and new files, even if it has to change the file names to have both. — Joe K, May 05 '14 at 19:59

score 1 · Answer 1 · answered Aug 13 '14 at 17:28

Distcp does not have that option. If you are using the Java API for it, it can be easily handled by checking if the destination path exist and changing the path in case it already exists. You can check that with a FileSystem object using the method exists(Path p).

Hadoop DistCp handle same file name by renaming

1 Answers1