How to change environment variables in mrjob for AWS accesskey and secretaccesskey

Question

How do I change the $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY in mrjob to enter my own credentials for AWS? I am using the Terminal on Mac OS X.

https://github.com/Yelp/mrjob

Thanks!

Answer 1

Well you have several ways. You could just change your environment variables called AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. You could probably do so using the setenv command, although I'm not a Mac OS expert.

A nice feature of mrjob is that it also reads a config called mrjob.conf and looks for it in default places such as your home directory, the /etc directory and the directory pointed to by the environment variable MRJOB_CONF. This config is in YAML format, so you should have a section like the following to setup your keys:

runners: emr: aws_access_key_id: XXXX aws_secret_access_key: YYYY ...

There are many more parameters you can setup in the config so I advise you look at an example config to see what you can do there.

I've also read that it works if you setup your keys in your boto config in the .boto file under either your home directory or the /etc directory, and you can add the following lines. Not tested myself, but should work as well:

aws_access_key_id = XXXX
aws_secret_access_key = YYYY