52

I am using 

hdfs dfs -put myfile mypath

and for some files I get 

put: 'myfile': File Exists
  • does that mean there is a file with the same name or does that mean the same exact file (size, content) is already there?
  • how can I specify an -overwrite option here?

Thanks!

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
ℕʘʘḆḽḘ
  • 18,566
  • 34
  • 128
  • 235

4 Answers4

66

put: 'myfile': File Exists

Means,the file named "myfile" already exists in hdfs. You cannot have multiple files of the same name in hdfs

You can overwrite it using hadoop fs -put -f /path_to_local /path_to_hdfs

Sumit
  • 1,360
  • 3
  • 16
  • 29
  • thanks! very helpful. i does put checks whether the transfer went through correctly? lets say i start hdfs put and the I close my console. does hdfs has now some corrupted file? – ℕʘʘḆḽḘ Apr 23 '16 at 21:34
  • 1
    To check whether copied file is correct (with respect to size) or not, you can use hdfs dfs -ls /filename. It will show the file size in bytes, along with other info. @ Noobie – Sumit Apr 23 '16 at 22:01
  • 4
    So `hdfs dfs -put -f input output` doesn't work? I've found sources that says it does – OneCricketeer Apr 24 '16 at 00:09
  • 2
    The accepted answer is wrong. I just tested hdfs dfs -put -f and it works perfectly. – Cos Aug 10 '17 at 07:19
  • 1
    In my case, I got an error `java.io.IOException: File -f does not exist.` – Xiwei Wang Aug 12 '20 at 09:00
42

You can overwrite your file in hdfs using -f command.For example 

hadoop fs -put -f <localfile> <hdfsDir>

OR

hadoop fs -copyFromLocal -f <localfile> <hdfsDir>

It worked fine for me. However -f command won't work in case of get or copyToLocal command. check this question

Community
  • 1
  • 1
salmanbw
  • 1,301
  • 2
  • 17
  • 23
5
  1. A file with the same name exists at the location you're trying to write to.
  2. You can overwrite by specifying the -f flag.
Pradeep Gollakota
  • 2,161
  • 16
  • 24
  • AFAIK, we cannot overwrite the files present in hdfs, and that is why hadoop is known for "write once read multiple" property. Am I wrong?? – Sumit Apr 24 '16 at 01:17
  • 1
    @user007, You can delete files and create a new file with the same name, overwrite. And in some cases you can append data at the end of the file but only at the end. Append is only available in hadoop version that include it and it is required for HBase and other framworks. – RojoSam Apr 24 '16 at 02:24
2

Just updates to this answer, in Hadoop 3.X the command a bit different 

hdfs dfs -put -f /local/to/path hdfs://localhost:9870/users/XXX/folder/folder2
MMSA
  • 810
  • 8
  • 22