how to write text to file in HDFS without appending

Asked Aug 19 '19 at 17:51

Active Aug 19 '19 at 18:03

Viewed 33 times

I wrote the following cli in order to write version to HDFS on remote machine

Note - The following CLI is also create the file - /user/version

echo $version 
0.21

ssh $remote_machine "su hdfs -c \"echo $version | hadoop fs -appendToFile - /user/version\" "

I give explain about the Cli

I am using ssh to remote machine in order to write the content of the variable - $version to /user/version ( while /user/version is HDFS file )

cli succeeded but this approach appending the version when cli run couple of times , and what we want is write the version without appending

example what we got ( on the remote machine )

su hdfs
$ hdfs dfs -cat /user/version
0.21
0.21
0.21

while the expected output should be

su hdfs
$ hdfs dfs -cat /user/version
0.21

summary:

cli should create the HDFS file - /user/version , and write the $version to file

edited Aug 19 '19 at 18:03

asked Aug 19 '19 at 17:51

jessica

Is `-appendToFile` right option? – Cyrus Aug 19 '19 at 18:08
no this isn't the right option but it close to what I want – jessica Aug 19 '19 at 18:10
Have you tried replacing `-appendToFile` with `-put`? – Cyrus Aug 19 '19 at 18:12
yes I did but I get - File exists – jessica Aug 19 '19 at 18:14
what I means is that when I run it in the first time its ok , but when I run it on the second time I get - File exists ( cli should be re-run ) – jessica Aug 19 '19 at 18:15
Have you tired replacing `-appendToFile` with `-copyFromLocal -f`? – Cyrus Aug 19 '19 at 18:16
no this approach is new for me , can you give example with full syntax? – jessica Aug 19 '19 at 18:16
as I understand - copyFromLocal means copy from current OS to HDFS , but this isnt what we need , we not need to copy local file to hdfs , only create new file in HDFS and file in HDFS should contain the #version – jessica Aug 19 '19 at 18:22
Have you tired `-put -f`? – Cyrus Aug 19 '19 at 18:28
2

yes now I try it , and this option works , great -:) ( the flag -f , do the different , its like force ) – jessica Aug 19 '19 at 18:31
you can post an answer and I will select your answer – jessica Aug 19 '19 at 18:33
I don't know anything about HDFS, I just know how to operate Google. – Cyrus Aug 19 '19 at 18:34
ok , so I will post the solution that you give me just for avryone that want to see – jessica Aug 19 '19 at 18:38

0 Answers0