0

I am trying to copy the NCDC Data from Amazon S3 to my local hadoop cluster by using following command.

hadoop distcp -Dfs.s3n.awsAccessKeyId='ABC' -Dfs.s3n.awsSecretAccessKey='XYZ' s3n://hadoopbook/ncdc/all input/ncdc/all

And getting error which is given below :

java.lang.IllegalArgumentException: AWS Secret Access Key must be specified as           the password of a s3n URL, or by setting the fs.s3n.awsSecretAccessKey property

Gone through the following question but of no big help.

Problem with Copying Local Data

Any hint about how to solve the problem . Detailed answer will be very appreciated for better understanding. Thanks

Community
  • 1
  • 1
LonelySoul
  • 1,212
  • 5
  • 18
  • 45

2 Answers2

1

Have you tried this:

Excerpt from AmazonS3 Wiki

Here is an example copying a nutch segment named 0070206153839-1998 at /user/nutch in hdfs to an S3 bucket named 'nutch' (Let the S3 AWS_ACCESS_KEY_ID be 123 and the S3 AWS_ACCESS_KEY_SECRET be 456):

% ${HADOOP_HOME}/bin/hadoop distcp hdfs://domU-12-31-33-00-02-DF:9001/user/nutch/0070206153839-1998 s3://123:456@nutch/

In you case, it should be something like this:

hadoop distcp s3n://ABC:XYZ@hadoopbook/ncdc/all hdfs://IPaddress:port/input/ncdc/all
Glorfindel
  • 21,988
  • 13
  • 81
  • 109
SSaikia_JtheRocker
  • 5,053
  • 1
  • 22
  • 41
0

You need to set up the aws id and password in the core-site.xml

<property>
<name>fs.s3n.awsAccessKeyId</name>
<value>xxxxxxx</value>
</property>
<property>
<name>fs.s3n.awsSecretAccessKey</name>
<value>xxxxxxxxx</value>
</property>

and restart your cluster

Snehan Solomon
  • 432
  • 3
  • 17