4

I am trying to use an S3 bucket as input data for my Elastic Map Reduce job flow. The S3 bucket does not belong to the same account as the EMR job flow. How and where should I specify the S3 bucket credentials to access the respective S3 bucket. I tried the following format:

s3n://<Access Key>:<Secret Key>@<BUCKET>

But it gives me the following error:

Exception in thread "main" java.lang.IllegalArgumentException: The bucket name parameter must be specified when listing objects in a bucket
at com.amazonaws.services.s3.AmazonS3Client.assertParameterNotNull(AmazonS3Client.java:2381)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:444)
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:785)
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.ensureBucketExists(Jets3tNativeFileSystemStore.java:80)
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:83)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.fs.s3native.$Proxy1.initialize(Unknown Source)
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:512)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1413)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:68)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1431)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:256)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:352)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:321)
at com.inmobi.appengage.emr.mapreduce.TestSession.main(TestSession.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:187)

How do I go about specifying the same?

Abhishek Jain
  • 4,478
  • 8
  • 34
  • 51

1 Answers1

3

You should try to add these credentials to the core-site.xml file. You can add the s3 credentials manually in the nodes or by using the boostrap action while launching the cluster.

You can launch the cluster with something like this:

ruby elastic-mapreduce --create --alive --plain-output --master-instance-type m1.xlarge --slave-instance-type m1.xlarge --num-instances 11 --name "My Super Cluster" --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop --args -c,fs.s3.awsAccessKeyId=< access-key >,-c,fs.s3.awsSecretAccessKey=< secret-key >

This should override the default values which are placed by EMR as per the account launching the cluster.

Amar
  • 11,930
  • 5
  • 50
  • 73
  • Did not work for me the args way. It says trying to execute /mnt/var/lib/bootstrap-actions/2/configure-hadoop --args -c,fs.s3n.awsAccessKeyId=,-c,fs.s3n.awsSecretAccessKey= in the bootstrapper logs and fails with the following error: 2013-08-26T08:47:22.715Z INFO Execution ended with ret val 255 2013-08-26T08:47:22.716Z ERROR Execution failed with code '255'. One more thing I am setting the S3 key options in "optional arguments" in configuring bootstrap actions from the JobFlow GUI. Is there any other way this could be made to work? – Abhishek Jain Aug 26 '13 at 09:56
  • Log into any of your slave nodes and see if the core-site.xml file in there has these changes or not ? Also try to launch cluster using the ruby cli provided by Amazon. – Amar Aug 26 '13 at 13:05
  • I was able to use the ruby cli and ran it as you've suggested above but I am getting a new error now: Exception in thread "main" Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 0504CE5FDBFC1D8B, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: . Can you help? – Abhishek Jain Aug 27 '13 at 09:57
  • Thanks for all the help. It was because the IAM credentials were screwed somewhere else your advice would have worked. – Abhishek Jain Aug 28 '13 at 12:08