s4cmd sync two buckets access denied

Question

I am trying to sync two s3 buckets:

s4cmd --dry-run sync s3://cgl-rnaseq-recompute-fixed/gtex s3://rnaseq.toil.20k/gtex

But I am getting the following error:

[Exception] An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
[Thread Failure] An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

The source bucket is publicly available. The second bucket is mine and I have access to it:

[centos@ip-172-30-3-12 data]$ s4cmd ls s3://rnaseq.toil.20k/
                 DIR s3://rnaseq.toil.20k/gtex/
                 DIR s3://rnaseq.toil.20k/pnoc/
                 DIR s3://rnaseq.toil.20k/target/
                 DIR s3://rnaseq.toil.20k/tcga/

Also I cannot ls on the source bucket using s4cmd but I can using s3cmd:

[centos@ip-172-30-3-12 data]$ s4cmd ls s3://cgl-rnaseq-recompute-fixed/gtex
[Exception] An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
[Thread Failure] An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

[centos@ip-172-30-3-12 data]$ s3cmd ls --requester-pays s3://cgl-rnaseq-recompute-fixed/gtex
                       DIR   s3://cgl-rnaseq-recompute-fixed/gtex/
2016-06-03 17:02    435553   s3://cgl-rnaseq-recompute-fixed/gtex-manifest

What could be going wrong? Any suggestions would be much appreciated.

You said you have access to that bucket? Can you paste what policy your account has for the S3? — Piyush Patil, Jul 19 '16 at 20:52
I don't know how to do that. (Don't know if it matters but I am not the admin). But I have showed in my question that I have access to both buckets by doing s3cmd ls. — Komal Rathi, Jul 19 '16 at 20:58
Are you able to go to the bucket to which you have access and see its content from the AWS Console? — Piyush Patil, Jul 19 '16 at 20:59
Yes I have showed that I can do ls on it but only when I use s3cmd with --requester-pays argument. — Komal Rathi, Jul 19 '16 at 21:03
Try this command and see if it works "s4cmd sync --acl-public s3://source-bucket/ s3://target-bucket/" — Piyush Patil, Jul 19 '16 at 22:24
Consider using the official aws-cli command set tools for S3: http://docs.aws.amazon.com/cli/latest/reference/s3/ In my case it work like a charm. — Conti, Jul 20 '16 at 07:33
@error2007s I get this error: `s4cmd.py: error: no such option: --acl-public` — Komal Rathi, Jul 20 '16 at 13:37
@Conti I was trying to use s4cmd because it is supposed to be faster in copying files between buckets. — Komal Rathi, Jul 20 '16 at 13:39
@error2007s I just know the s3 bucket is `s3://cgl-rnaseq-recompute-fixed/gtex` and it is publicly accessible. — Komal Rathi, Jul 20 '16 at 14:05

score 0 · Accepted Answer · answered Jul 20 '16 at 15:42

To achieve the s3cmd behavior, use wildcards:

s4cmd sync s3://bucket/path/dirA/* s3://bucket/path/dirB/

Note s4cmd doesn't support dirA without trailing slash indicating dirA/* as what rsync supported.

So in you case you have to use.

s4cmd --dry-run sync s3://cgl-rnaseq-recompute-fixed/gtex/* s3://rnaseq.toil.20k/gtex

Check this documentation for s4cmd it is very helpful.

https://github.com/bloomreach/s4cmd

s4cmd sync two buckets access denied

1 Answers1