Hadoop difference between put and copyfromlocal

Question

CopyFromLocal will upload data from local filesystem.

Do put will upload data from any fs eg. Local fs, amazon s3 Or only from local fs ???

Possible duplicate of [Difference between hadoop fs -put and hadoop fs -copyFromLocal](https://stackoverflow.com/questions/7811284/difference-between-hadoop-fs-put-and-hadoop-fs-copyfromlocal) — OneCricketeer, Sep 12 '18 at 12:48
If you're using S3 or other Hadoop filesystem, you'd use DistCp — OneCricketeer, Sep 12 '18 at 12:48
There is only one difference between put and copyfromlocal, that is. Copyfromlocal copy data only from local and. Put will copy from different filesystem, and i am looking for an answer what are those filesystem. Please someone give all the filesystem names which can get/put data using "hadoop fs-put" command — Jayaprakash, Sep 12 '18 at 16:43
You cannot `get` data with `put`. That doesn't make sense. The source code of `put` and `copyFromLocal` are using the exact same Java methods. As I mentioned, DistCp does a copy between different filesystems — OneCricketeer, Sep 12 '18 at 19:36

score 0 · Answer 1 · answered Sep 12 '18 at 18:21

Please find the usage of both commands.

put
=======
Usage: hadoop fs -put [-f] [-p] [-l] [-d] [ - | <localsrc1> .. ]. <dst>

Copy single src, or multiple srcs from local file system to the destination file system. Also reads input from stdin and writes to destination file system if the source is set to “-”

Copying fails if the file already exists, unless the -f flag is given.

Options:

-p : Preserves access and modification times, ownership and the permissions. (assuming the permissions can be propagated across filesystems)
-f : Overwrites the destination if it already exists.
-l : Allow DataNode to lazily persist the file to disk, Forces a replication factor of 1. This flag will result in reduced durability. Use with care.
-d : Skip creation of temporary file with the suffix ._COPYING_.
Examples:

hadoop fs -put localfile /user/hadoop/hadoopfile
hadoop fs -put -f localfile1 localfile2 /user/hadoop/hadoopdir
hadoop fs -put -d localfile hdfs://nn.example.com/hadoop/hadoopfile
hadoop fs -put - hdfs://nn.example.com/hadoop/hadoopfile Reads the input from stdin.
Exit Code:

Returns 0 on success and -1 on error.

copyFromLocal
=============

Usage: hadoop fs -copyFromLocal <localsrc> URI

Similar to the fs -put command, except that the source is restricted to a local file reference.

Options:

-p : Preserves access and modification times, ownership and the permissions. (assuming the permissions can be propagated across filesystems)
-f : Overwrites the destination if it already exists.
-l : Allow DataNode to lazily persist the file to disk, Forces a replication factor of 1. This flag will result in reduced durability. Use with care.
-d : Skip creation of temporary file with the suffix ._COPYING_.

This is what the docs say, but the source code says otherwise — OneCricketeer, Sep 12 '18 at 19:59

OneCricketeer · Answer 2 · 2018-09-12T20:00:21.727

They are identical (as per extends Put, plus no other methods or conditionals otherwise). And as of this answer, that CopyCommands.java class has not had new commits since 2014.

  public static class CopyFromLocal extends Put {
        public static final String NAME = "copyFromLocal";
        public static final String USAGE = Put.USAGE;
        public static final String DESCRIPTION = "Identical to the -put command.";
  }

Please someone give all the filesystem names which can get/put data using "hadoop fs-put" command

Any Hadoop Compatible FileSystem (HCFS) is supported using hadoop fs commands. Only HDFS is supported by hdfs dfs

Hadoop difference between put and copyfromlocal

2 Answers2