Recommended way to manage credentials with multiple AWS accounts?

Question

What is the best way to manage multiple Amazon Web Services (AWS) accounts through boto?

I am familiar with BotoConfig files, which I'm using. But each file describes only a single account...and I am working with more than just the one organization. For all the usual legal, financial, and security reasons, those accounts cannot be commingled.

Currently I am using one boto config file per account. E.g.:

• ~/.boto default account
• ~/.boto_clowncollege for "clowncollege" account
• ~/.boto_razorassoc for "razorassoc" account
• ~/.boto_xyz for "xyz" account

Then something like:

def boto_config_path(account=None):
    """
    Given an account name, return the path to the corresponding boto
    configuration file. If no account given, return the default config file.
    """
    path = '~/.boto' + ('_' + account if account else '')
    clean_path = os.path.abspath(os.path.expanduser(path))
    if os.path.isfile(clean_path):
        return clean_path
    else:
        errmsg = "cannot find boto config file {} for {}".format(clean_path, account)
        raise ValueError(errmsg)

def aws_credentials(account=None):
    """
    Return a tuple of AWS credentials (access key id and secret access key) for
    the given account.
    """
    try:
        cfg = INIConfig(open(boto_config_path(account)))
        return ( cfg.Credentials.aws_access_key_id, cfg.Credentials.aws_secret_access_key )
    except Exception:
        raise

conn = EC2Connection(*aws_credentials('razorassoc'))

Good, bad, or indifferent? Suggested improvements?

Answer 1

updated 2015-02-06, corrected 2015-03-19 by following top section

New standardized sharing of boto and AWSCLI credentials (boto>==2.29.0)

Since boto 2.29 there is new easy way for sharing BOTO and AWS CLI credentials as described by Mike Garnaat in A New and Standardized Way to Manage Credentials in the AWS SDKs

The aim is to:

1. allow sharing credentials by boto, AWSCLI and possibly other SDKs
2. keep all configuration in single file, located in user profile directory
3. allow using named profiles
4. keep it as simple as possible (e.g. prevent conflicts with other methods)

Create credentials file

Create the file ~/.aws/credentials (Mac/Linux) or %USERPROFILE%\.aws\credentials (Windwos) as follows:

[default]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr
region = eu-west-1

[jekyl]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr
region = eu-west-1

[hyde]
aws_access_key_id = AxxxZ
aws_secret_access_key = CxxxZ
region = eu-west-1

From now on, you may use a code like this:

Use default profile

import boto
con = boto.connect_s3()

Use explicit profile set by AWS_PROFILE env. var

(this is my favourite option keeping profile name out of code and still giving the deployer of my application a chance to pick specific profile)

$ export AWS_PROFILE=jekyl

and keep your code as simple as before:

import boto
con = boto.connect_s3()

Specify explicit profile in your code

import boto
con = boto.connect_s3(profile_name="jekyl")

This is all you typically need to do

The logic for picking proper credentials is described in boto issue #2292 as follows:

The loading order from highest to lowest precedence:

1.Directly passed from code

2. Environment variables for key/secret

3. Environment variables for profile

4. Shared credential file explicit profile

5. Shared credential file default profile

6. Config file explicit profile

7. Config file Credentials section

A profile passed from code overrides any set in an environment variable.

To keep things clean and simple, it is good to get rid of older methods, so remove any old style files (like ~/.aws/config or ~/.boto), unset environmental varialbe BOTO_CONFIG if set and possibly also the file, to which such variable points to.

And that is really all for boto >=2.29.0

Note: Do not attempt to control location of config file by env.variable (like AWS_CONFIG_FILE), it does not work as expected.

Use boto config profile (boto>=2.24.0)

Following description is kept here only for those, who cannot upgrade to boto 2.29.0 or higher

Since boto 2.24.0 there is a feature called profile_name

In your ~/.boto file you already have your [Credentials] section, this will serve as fallback option, and then [profile ] sections serving for different profiles:

[Credentials]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr

[profile jekyl]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr

[profile hyde]
aws_access_key_id = AxxxZ
aws_secret_access_key = CxxxZ

Then, when creating connection, you use this way:

import boto
con = boto.connect_s3(profile_name="jekyl")

Note, that this feature is available since boto 2.24.0.

Tutorial is here http://docs.pythonboto.org/en/latest/boto_config_tut.html?highlight=profile

There are even some notes about using keyrings, but I will first get used to this profile stuff, which I was dreaming of few years.

Sharing config file with AWSCLI

AWSCLI became really great tool. As format of config file is almost the same, I use it in following way:

1. keep ~/.aws/config file as created by AWSCLI (this is default location)
2. copy section [default] and rename it to [Credentials] (leaving the same values inside).
3. add whatever profiles I use
4. set BOTO_CONFIG variable to point to this ~/.aws/config file.

The ~/.boto would then become `~/.aws/config with following content:

[default]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr

[Credentials]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr

[profile jekyl]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr

[profile hyde]
aws_access_key_id = AxxxZ
aws_secret_access_key = CxxxZ

This way, it gets shared for both AWSCLI and boto including profiles.

Answer 2

In the future, boto will provide better tools to help you manage multiple credentials but at the moment, there are a couple of environment variables that might help out.

First, you can set BOTO_CONFIG to point to a boto config file that you want to use and it will override any config file found in the normal locations.

Secondly, you can set BOTO_PATH to a colon-separated list of places to look for a boto config file and it will search there first, prior to the normal search locations.

Neither of those give you exactly what you want but it may make it easier to accomplish with a bit less code.

If you have ideas about how you would like this to work in boto, please let me know!

Answer 3

Rather than creating a bunch of separate boto config files, consider using the ConfigParser module and creating a section in the .boto file for each of your accounts.

your .boto file might look something like this

#Contents of ~/.boto
[clown-college]
aws_access_key_id = 123sesamestreet
aws_secret_access_key = 678idsaf567ujd
[razor-assoc]
aws_access_key_id = 437piedmont
aws_secret_access_key = 997567ujdfs

In your python code, use ConfigParser to load the appropriate access key for the account you wish to use.

import ConfigParser
from os.path import expanduser

########## BEGIN MAIN ##############

# get the path to the user's homedir
user_home = expanduser("~")

#load their .boto config file
config = ConfigParser.ConfigParser()
config.read([str(user_home + "/.boto")])

#get the keypair for ClownCollege
print config.get('clown-college', 'aws_access_key_id')
print config.get('clown-college', 'aws_secret_access_key')

print config.get('razor-assoc', 'aws_access_key_id')
print config.get('razor-assoc', 'aws_secret_access_key')

This can be wrapped in a function to use across your boto code to easily set the correct account.

Answer 4

As of boto >= 2.38, it seems like all previous solutions can cause immense headaches and problems.

After testing this extensively today on multiple BSD based platforms, it appears that the most effective way now to manage the different profile authentications for AWSCLI and py-boto is to use the aws configure interactive script. Call it without a profile, and it will fill out a [default] block for both the .aws/config and .aws/credentials files, and in addition it will configure whatever other magic it needs to get boto working with your aws toolset (although it's unclear yet just what hexes it's casting on your localhost). Call it again with any profile name, and it will create an appropriate entry.

Note that this still doesn't work for versions of boto < 2.3.

aws configure --profile somename
AWS Access Key ID [None]: XXXXXXXXXXXXXXXXXXXX
AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Default region name [None]: us-east-1
Default output format [None]: json

Answer 5

Note that per http://boto.cloudhackers.com/en/latest/boto_config_tut.html -- when using connect_cloudwatch() for boto version 2.48.0 I actually needed to provide the

cloudwatch_region_endpoint

Or the connection still would use the same region as the current instance, when pushing data from the instance itself in EC2 (when the instance is in a different region as where it's pushing to).

cat .boto
[Boto]
cloudwatch_region_name = us-west-2
cloudwatch_region_endpoint = monitoring.us-west-2.amazonaws.com

I tried adding that to the .aws/credentials profile, but it didn't seem to work.

cat .aws/credentials 
[cloudwatch_centralized_reporting]
region = us-west-2
aws_access_key_id = XXX
aws_secret_access_key = XXX