1

I'm adapting a bash script to python. The script invokes the aws cli to download a file from s3:

aws s3 cp s3://some_bucket/some/key /some/path

I could of course call aws from python, but it feels better to use a library like boto to do it; I can customize the operations more and exceptions raised will be more specific than CalledProcessError, etc. So I wrote:

s3 = boto.connect_s3()
bucket = s3.get_bucket("some_bucket")
key = bucket.get_key("some/key")
contents = key.get_contents_as_string().decode("utf-8").strip()
path = "/some/path"
with open(path, "w") as f:
    f.write(contents + "\n")

Although this works on my desktop (which authorizes with access keys), it does not work on the ec2 instance I need it to run on (which uses IAM). Instead, it hangs at the get_bucket() call. I would think this is due to IAM permissions, but the above aws s3 cp command works fine. I tried to look through the source of the aws CLI, but it's quite complicated and it doesn't seem to be using boto.

What would cause the two to act differently, and is there a way that I can adapt my boto usage such that it works the same as the CLI?

limp_chimp
  • 13,475
  • 17
  • 66
  • 105
  • Using Boto, are you able to run any other commands with s3, such as lists_buckets? Or none of them work? – filipebarretto May 03 '17 at 16:50
  • So, does the AWS CLI work on the EC2 instance? If so, it is definitely using credentials that have access to the S3 bucket. Here's how the AWS CLI looks for credentials: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html. And this is how boto looks for them: http://stackoverflow.com/a/21345540/347777 There is got to be a mismatch on the credentials if boto and CLI work on your desktop, but only the CLI works on EC2. – Viccari May 03 '17 at 18:08
  • Check your region setting in your credentials file on the EC2 instance. I often see people put `us-west-2a` as a Region, where is should only be `us-west-2`. This causes boto/CLI to try to connect to a non-existent endpoint, which takes a while to timeout. You can check the configuration by running `aws configure` on the instance and checking the region. – John Rotenstein May 04 '17 at 00:41
  • BTW, you can use `get_contents_to_file()` to download directly to a file. Also, these days you should try to use [boto3](boto3.readthedocs.org) rather than boto. – John Rotenstein May 04 '17 at 00:42

0 Answers0