57

I'm trying to copy files locally from s3 bucket. I can get the list of files on my bucket:

aws s3 ls  s3://myBucket/myDirectory/todaysFiles/

But when I try to copy the files locally:

aws s3 cp s3://myBucket/myDirectory/todaysFiles/ .

I get this error:

fatal error: An error occurred (404) when calling the HeadObject operation: Key "myDirectory/todaysFiles/" does not exist

But I try to copy just one file locally:

 aws s3 cp s3://myBucket/myDirectory/todaysFiles/somefile .

I get this error:

 warning: Skipping file s3://myBucket/myDirectory/todaysFiles/somefile. Object is of storage class GLACIER. Unable to perform download operations on GLACIER objects. You must restore the object to be able to the perform operation. See aws s3 download help for additional parameter options to ignore or force these transfers.

Any of you knows why I'm getting this error or way around this errors?

I really appreciate your help

user2924482
  • 8,380
  • 23
  • 89
  • 173

4 Answers4

102

For the first error - add the recursive flag:

aws s3 cp s3://myBucket/myDirectory/todaysFiles/ . --recursive

This will copy all the files in the "todaysFiles" directory to the current directory.

However, the second error indicates that your files are in Glacier. This complicates things a bit as Glacier is not real time - depending on what you're willing to pay it can be hours before the data is restored. See the Restoring Objects docs for a bit more information. You can't copy from S3 until the object are restored from Glacier to S3.

Note that if you do this you will have costs from both Glacier and S3.

As an aside, if these files are really files from today there should be a much longer time between storage on S3 and the push to Glacier. But I'm guessing that the parent directories may have a date related component too.

stdunbar
  • 16,263
  • 11
  • 31
  • 53
11

Use --recursive at the end if the Source contains multiple folders.

aws s3 cp s3Uri Local --recursive
Saman
  • 333
  • 3
  • 9
0

In case it helps anyone, you can also use

aws s3 sync s3://<bucketname>/<folder>/ ./<folder>

This should work without having to use the recursive. It has the benefit that it skips files already present locally.

As mentioned by @raju, glacier is a cheap storage system on AWS, meaning if you want to retrieve the data you first need to put a request in to fetch the data before you can download it. It's made to be cheap for long term storage that isn't accessed frequently.

MrFronk
  • 382
  • 5
  • 23
-3
pip install --upgrade --user awscli

aws s3 cp s3://inbound/test/ /inbound/ --recursive --region us-west-2
K.Dᴀᴠɪs
  • 9,945
  • 11
  • 33
  • 43
raju
  • 129
  • 1
  • 9
  • The `--region` switch applies to the destination bucket, which is meaningless here because it's the local file system. To specify the region of the source bucket you use the `--source-region` switch instead. REF: https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html – AlwaysLearning Jul 11 '19 at 23:43
  • Looks like OP already had awscli installed, no need to suggest it. – Luigi Lopez Nov 22 '20 at 05:16
  • s3 is global isn't it? what's the point of specifying the region? – DJ_Stuffy_K Mar 12 '21 at 05:08