Why anonymous FTP to HDFS DistCp does not work?

Question

I tried below command to copy FTP data to HDFS,

 hadoop distcp ftp://ftp.ncdc.noaa.gov/pub/data/noaa/1901/ data/noaa/1901/

Here is the error I received

15/03/04 11:29:13 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[ftp://ftp.ncdc.noaa.gov/pub/data/noaa/1901], targetPath=data/noaa/1901, targetPathExists=false, preserveRawXattrs=false}
15/03/04 11:29:13 INFO client.RMProxy: Connecting to ResourceManager at m-hadoop-master.c.swift-influence-86909.internal/10.240.143.253:8032
15/03/04 11:29:14 ERROR tools.DistCp: Exception encountered 
java.io.IOException: Login failed on server - ftp.ncdc.noaa.gov, port - 21 as user 'null'
    at org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.java:144)
    at org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSystem.java:405)
    at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
    at org.apache.hadoop.fs.Globber.glob(Globber.java:252)
    at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1625)
    at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
    at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:84)
    at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:353)
    at org.apache.hadoop.tools.DistCp.execute(DistCp.java:160)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)

score 0 · Answer 1 · answered Mar 04 '15 at 13:32

You have to fill in the username, even for "anonymous" login.

The Apache FTPClient (used internally FTPFileSystem) always sends the USER command to the FTP server. So it needs some username to use along.

Use username "anonymous". That's a common practice. Most FTP clients, when instructed to use "anonymous" login, will internally use "anonymous" username automatically.

If a password in required, use one a form of email address ("user@example.com").

Why anonymous FTP to HDFS DistCp does not work?

1 Answers1