24

I'm trying to use AWS' recently announced "IAM roles for EC2" feature, which lets security credentials automatically get delivered to EC2 instances. (see http://aws.amazon.com/about-aws/whats-new/2012/06/11/Announcing-IAM-Roles-for-EC2-instances/).

I've set up an instance with an IAM role as described. I can also get (seemingly) proper access key / credentials with curl.

However, boto fails to do a simple call like "get_all_buckets", even though I've turned on ALL S3 permissions for the role.

The error I get is "The AWS Access Key Id you provided does not exist in our records"

However, the access key listed in the error matches the one I get from curl.

Here is the failing script, run on an EC2 instance with an IAM role attached that gives all S3 permissions:

import urllib2
import ast
from boto.s3.connection import S3Connection

resp=urllib2.urlopen('http://169.254.169.254/latest/meta-data/iam/security-credentials/DatabaseApp').read()
resp=ast.literal_eval(resp)
print "access:" + resp['AccessKeyId']
print "secret:" + resp['SecretAccessKey']
conn = S3Connection(resp['AccessKeyId'], resp['SecretAccessKey'])
rs= conn.get_all_buckets()
Nils
  • 5,612
  • 4
  • 34
  • 37

2 Answers2

61

If you are using boto 2.5.1 or later it's actually much easier than this. Boto will automatically find the credentials in the instance metadata for you and use them as long as no other credentials are found in environment variables or in a boto config file. So, you should be able to simply do this on the EC2 instance:

>>> import boto
>>> c = boto.connect_s3()
>>> rs = c.get_all_buckets()

The reason that your manual approach is failing is that the credentials associated with the IAM Role is a temporary session credential and consists of an access_key, a secret_key and a security_token and you need to supply all three of those values to the S3Connection constructor.

Radek Simko
  • 15,886
  • 17
  • 69
  • 107
garnaat
  • 44,310
  • 7
  • 123
  • 103
  • For the manual approach, you're right, adding the security_token fixed it. Thank you! I marked you as the 'answer' for that. – Nils Jun 21 '12 at 02:58
  • The simplified version you posted also works after upgrading to 2.5.1. So if I could give you 2 accepts, I would! – Nils Jun 21 '12 at 02:58
  • Great, glad it's working for you now. I like the new IAM Roles feature, it's really slick. – garnaat Jun 21 '12 at 03:08
  • Boto v2.5.2 does not yet support Key.generate_url() using EC2/IAM Roles. It results in an invalid querystring. But the 'develop' branch (as of 7/24/2012) does support roles and can be installed using pip: 'pip install git+git://github.com/boto/boto.git@develop' – Jonathan Oliver Jul 24 '12 at 18:49
  • Just make sure you have the correct boto version, get it from Github on develp branch or something. – Ali Feb 24 '13 at 16:17
  • Versions 2.6, 2.7, and 2.8 of boto all support IAM Roles and the generation of signed S3 query strings using IAM Roles. – garnaat Feb 24 '13 at 16:28
5

I don't know if this answer will help anyone but I was getting the same error, I had to solve my problem a little differently. First, my amazon instance did not have any IAM roles. I thought I could just use the access key and the secret key but I kept getting this error with only those two keys. I read I needed a security token as well, but I didn't have one because I didn't have any IAM roles. This is what I did to correct the issue:

  1. Create an IAM role with AmazonS3FullAccess permissions.
  2. Start a new instance and attach my newly created role.

  3. Even after doing this it still didn't work. I had to also connect to the proper region with the code below:

    import boto.s3.connection
    conn = boto.s3.connect_to_region('your-region')
    conn.get_all_buckets()

Gabriel
  • 624
  • 1
  • 7
  • 20