7

I'm trying to use a log rotation configuration for my nginx server that I'm using as a reverse proxy machine located on an EC2 Ubuntu instance.

I want to store those logs on a S3 bucket after a rotation but I'm only getting "access denied, are you sure you keys have ListAllMyBuckets permissions errors" when I'm trying to configure s3cmd tools.

I'm pretty sure that my credentials is correctly configured at IAM, tried at least five different credentials (even the root cred) with the same result. It works fine to list all of my buckets from my local computer with aws cli tools with the same credentials so it puzzles me that I don't have any access just on my EC2 instance.

this is what I run:

which s3cmd
/usr/local/bin/s3cmd

s3cmd --configure --debug

Access Key: **************
Secret Key: *******************************
Encryption password: 
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

and this is the result 

...
DEBUG: ConnMan.put(): connection put back to pool (http://s3.amazonaws.com#1)
DEBUG: S3Error: 403 (Forbidden)
DEBUG: HttpHeader: x-amz-id-2: nMI8DF+............
DEBUG: HttpHeader: server: AmazonS3
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: x-amz-request-id: 5912737605BB776C
DEBUG: HttpHeader: date: Wed, 23 Apr 2014 13:16:53 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'AccessDenied'
DEBUG: ErrorXML: Message: 'Access Denied'
DEBUG: ErrorXML: RequestId: '5912737605BB776C'
DEBUG: ErrorXML: HostId: 'nMI8DF+............
ERROR: Test failed: 403 (AccessDenied): Access Denied
ERROR: Are you sure your keys have ListAllMyBuckets permissions?

The only thing that is in front of my nginx server is a load balancer, but I can't see why it could interfere with my request. Could it be something else that I've missed?

Sam
  • 418
  • 2
  • 6
  • 18

6 Answers6

9

Please check That IAM user permission which keys you are using

Steps would be

  • AWS console go to IAM panel
  • IAM user > Select that User > in the bottom menu 2nd tab is permission

  • attach a user policy

    {
"Version": "2012-10-17",
"Statement": [
 {
  "Effect": "Allow",
  "Action": ["s3:ListAllMyBuckets"],
  "Resource": "arn:aws:s3:::*"
},
{
  "Effect": "Allow",
  "Action": [
    "s3:ListBucket",
    "s3:GetBucketLocation"
  ],
  "Resource": "arn:aws:s3:::YOU-Bucket-Name"

},
{
  "Effect": "Allow",
  "Action": [
    "s3:PutObject",
    "s3:GetObject",
    "s3:DeleteObject"
  ],
  "Resource": "arn:aws:s3:::YOU-Bucket-Name/*"


}
]
}

Let me know how it goes

prayagupa
  • 30,204
  • 14
  • 155
  • 192
abaid778
  • 1,101
  • 1
  • 11
  • 24
  • Still no result, I even tested the new permission through IAM simulate policy website where it worked out just fine to list all buckets. – Sam Apr 23 '14 at 14:17
  • can you paste your user policy? – abaid778 Apr 23 '14 at 14:31
  • { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": "*" } ] } that's the group policy that I'm using, plus the user policy that you wrote – Sam Apr 23 '14 at 14:47
  • Please add full policy I have same issue when I added these full policy that works { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:ListAllMyBuckets"], "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::YOU-Bucket-Name" }, "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::YOU-Bucket-Name/*" } ] } – abaid778 Apr 23 '14 at 16:32
  • I tried now with {"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": ["s3:ListAllMyBuckets"],"Resource":"arn:aws:s3:::*"},{"Effect":"Allow","Action": ["s3:ListBucket","s3:GetBucketLocation"],"Resource": "arn:aws:s3:::pdx-logging"},{"Effect": "Allow","Action": [ "s3:PutObject","s3:GetObject","s3:DeleteObject"],"Resource": "arn:aws:s3:::pdx-logging/*"}]} and still getting the same result, even tried this with the other solutions that is posted here without any success. – Sam Apr 24 '14 at 08:00
  • 1
    Thanks this helped me with my S3 permission. – Sudipta Basak Oct 08 '15 at 04:36
9

Please dont trust the --configure switch:

i was facing the same problem. it was showing 403 in --configure but at the end i saved the Settings and then tried:

ERROR: Test failed: 403 (AccessDenied): Access Denied
Retry configuration? [Y/n] n
Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

# s3cmd put MyFile s3://MyBucket/

& it worked..

Mr. Pundir
  • 549
  • 6
  • 10
  • Same here - thanks for posting that, saved me a bunch of time trying to work out why it was apparently broken. – Phil Gyford May 31 '15 at 16:26
  • This usually happens when you have access to internal directory of bucket but not the root directory. – Mr. Pundir Jun 30 '15 at 13:51
  • I'm not sure I understand the explanation here. Once `Configuration saved to '/root/.s3cfg'` is displayed, what do you need to do? – Brosef Sep 19 '16 at 01:24
6

s3cmd creates a file called .s3cfg in your home directory when you set this up. I would make sure you put this file somewhere where your logrotate script can read this, and use the -c flag.

For example to upload the logfile.txt file to the logbucket bucket:

/usr/local/bin/s3cmd -c /home/ubuntu/.s3cfg put logfile.txt s3://logbucket

user3566750
  • 136
  • 3
1

What is the version of s3cmd you are using?

I tried it using s3cmd 1.1, it seems s3cmd 1.1 does not work with IAM roles.

But someone says s3cmd 1.5 alpha2 has support for IAM roles.(http://t1983.file-systems-s3-s3tools.file-systemstalk.info/s3cmd-1-5-0-alpha2-iam-roles-supportincluded-t1983.html)

I have tried s3cmd 1.5 beta1(https://github.com/s3tools/s3cmd/archive/v1.5.0-beta1.tar.gz), it works fine with IAM roles.

So there are two ways to access s3 bucket of s3cmd:

  1. Using access key and secret key `

    you need to set a config file in /root/.s3cfg(default path) as bellow

    access_key=xxxxxxxx secret_key=xxxxxxxxxxxxxxxxxxxx

    Note that just set above two key-value in .s3cfg, no need other keys.

    `
  2. Using IAM add s3 policy with s3cmd > 1.5 alph2. `

    you need add a IAM to ec2 instance, this role may has a policy as bellow

    { "Effect": "Allow", "Action": [ "s3:" ], "Resource": "" } `
Adam Parkin
  • 17,891
  • 17
  • 66
  • 87
user3567113
  • 11
  • 2
  • I'm using version 1.0.0-1, I'll try to update to 1.5 too and see if this solves my problems. – Sam Apr 24 '14 at 07:36
  • I think the root credentials must can access s3 buckets. You said that you can use it to list all s3 buckets with aws cli tool in local machine.
    But have you tried it with s3cmd in local machine?

    During configuring s3cmd credentials, there is a step that:
    `Test access with supplied credentials? [Y/n] y
    Please wait, attempting to list all buckets...`
    Can you get the bucket list using the root credential here?     – user3567113 Apr 26 '14 at 04:07
0

I found out a solution for my problems by deleting all installation of s3cmd. Then made sure that apt-get was up to date and installing it from apt-get again. After my configuration (the same as before) it worked out just fine!

Sam
  • 418
  • 2
  • 6
  • 18
0

I also had a similar problem. Even after associating my EC2 instance to an IAM role with s3 full access policy, my s3cmd was failing as there wasn't any .s3cfg file in it. I fixed by updating the version of my s3cmd.

sudo pip install s3cmd==1.6.1

Did the trick!

Selvaram G
  • 727
  • 5
  • 18