How to copy the output of -text HDFS command into another file?

Question

Is there any way we can copy text content of hdfs file into another file system using HDFS command:

 hadoop fs -text /user/dir1/abc.txt

Can I print the output of -text into another file by using -cat or any method ?:

 hadoop fs -cat /user/deepak/dir1/abc.txt 

Answer 1

As it's written in the documentation you can use hadoop fs -cp to copy files in hdfs. You can use hadoop fs -copyToLocal to copy files from hdfs to local file system. If you want to copy files from one hdfs to another then use DistCp tool.

Answer 2

As a general command line tip you can use | to another program or > or >> to a file, e.g.

# Will output to standard output (console) and the file /my/local/file
# this will overwrite the file, use ... tee -a ... to append
hdfs dfs -text /path/to/file | tee /my/local/file

# Will redirect output to some other command
hdfs dfs -text /path/to/file | some-other-command

# Will overwrite /my/local/file
hdfs dfs -text /path/to/file > /my/local/file

# Will append to /my/local/file
hdfs dfs -text /path/to/file >> /my/local/file

Answer 3

Thank you I did use streaming jar example in hadoop-home lib folder as follow :

hadoop -jar hadoop-streaming.jar -input hdfs://namenode:port/path/to/sequencefile \
-output /path/to/newfile -mapper "/bin/cat" -reducer "/bin/cat" \
-file "/bin/cat" -file "/bin/cat" \
-inputformat SequenceFileAsTextInputFormat

you can use "/bin/wc" in case you would like to count the number of lines at the hdfs sequence file.

Answer 4

you can use following:

1. copyToLocal
   hadoop dfs -copyToLocal /HDFS/file /user/deepak/dir1/abc.txt
2. getmerge
   hadoop dfs -getmerge /HDFS/file /user/deepak/dir1/abc.txt
3. get
   hadoop dfs -get /HDFS/file /user/deepak/dir1/abc.txt