1

One of my EMR job relies on getting the AWS access key id and secret access key from the fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey properties, respectively. However, when I run EMR cluster using the default EC2 and EMR roles, those properties do not exist.

How do I get these access keys while running the EMR clusters with the default roles, seeing that using IAM roles is a requirement at this point? One way I could do it is set these properties explicitly myself, but I want to see if I missed something else.

Kiet Tran
  • 1,458
  • 2
  • 13
  • 22

1 Answers1

2

Using roles and not hard coded keys is a best practice (http://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html). An example of this on EMR is the underlying Hadoop FS calls use the role assigned to the EC2 instance in order to generate temporary security credentials.

Your application can be built to do the same (http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-usingrole-ec2instance.html) such that it fetches a temporary access key and secret with token from the assigned role instead of reading it from a fixed config.

ChristopherB
  • 2,038
  • 14
  • 18
  • The problem I have with this is that my EMR job uses AmazonS3EncryptionClient, and I need to initialize it with an `AWSCredentials` object or it would make anonymous request. I haven't found a way to initialize `AWSCredentials` with just IAM role, and since it's a role and not a user, I cannot generate an access key/secret key for it. – Kiet Tran Jul 22 '15 at 19:13
  • Can you use a different constructor using a credentials provider? See http://docs.aws.amazon.com/AmazonS3/latest/dev/encrypt-client-side-asymmetric-master-key.html and http://docs.aws.amazon.com/AWSSdkDocsJava/latest//DeveloperGuide/credentials.html – ChristopherB Jul 22 '15 at 21:20
  • It looks like I can use a `DefaultAwsCredentialsProviderChain` since it seems to use the instance profile credentials from EC2. Using `InstanceProfileCredentialsProvider` would also work. – Kiet Tran Jul 23 '15 at 02:19