46

I am trying download a directory inside s3 bucket. I am trying to use transfer to download a directory from S3 bucket but I am getting an error as "An error occurred (404) when calling the HeadObject operation: Not Found". Please help.

S3 structure:
  **Bucket
     Folder1
        File1**

Note: Trying to download Folder1

transfer.download_file(self.bucket_name, self.dir_name, self.file_dir + self.dir_name)
Jack
  • 957
  • 3
  • 10
  • 23
  • 2
    You can't download a "folder." You download *objects*, individually. – Michael - sqlbot Oct 08 '17 at 20:52
  • To add to Michael's comment, just because there is an object at animals/dogs/boxer.png does not mean that there's an object (representing a folder) at either animals or at animals/dogs so HeadObject would typically yield 404 for both of those. – jarmod Oct 08 '17 at 22:05
  • S3 is an object storage. The path you see is actually part of the object name. To grab all object under the same "path" , you must specify the "PREFIX" parameter. – mootmoot Oct 11 '17 at 08:49

4 Answers4

65

I had the same issue recently. You are probably misspelling the path and folder name. In my case, for example, I was messing up with the '/'.

To fix it, make sure the variables you are using as arguments for the function contains the correct names of the directories, folders and files as it is in S3. Also, make sure you put the '/' in the correct places in the correct variables. For instance, in my case I found that:

  • bucket name: bucket_name (with no '/' in the end, and no 's3://')
  • directory name: folder1/folder2/file_name (with no '/' in the beginning)

I hope it helps you and others to get around this error easily.

Marcus Vinicius Melo
  • 1,118
  • 16
  • 23
  • My path and folder name are fine. When I run it using my local folder I see the files in by bucket. However I get this error when I deploy it on my Digital Ocean App Platform. – Yousuf Farhan Aug 19 '21 at 23:38
  • For future readers, also note that you get this error if the path itself is wrong/broken or you give folder path instead of the file path. – Pramesh Bajracharya Sep 17 '21 at 05:47
2

Yet another possibly is that you entered an incorrect endpoint_url parameter when creating your S3 resource.

For future users, create your resource like this:

s3 = boto3.resource(
  's3',
  region_name=[your region, e.g. eu-central-1],
  aws_access_key_id=[your access key],
  aws_secret_access_key=[your secret key]
)

In the above, it is possible to pass an endpoint_url, as I erroneously did (I later found out that I had accidentally passed the endpoint URL to a different AWS service).

If you are using AWS CLI in order to authenticate, you can omit the region_name, aws_access_key, and aws_secret_access_key parameters, like so:

s3 = boto3.resource('s3')
Shreyas
  • 667
  • 2
  • 7
  • 20
0

Spent much time finding why I got this error message with DigitalOcean platform.

The request should be performed as that:

client = boto3.client('s3', endpoint_url='https://fra1.digitaloceanspaces.com')
client.download_file('mybucketname', 'remotefilekeytoread', 'localfilenametosave')

If endpoint_url is set as 'https://mybucketname.fra1.digitaloceanspaces.com', download will fail with 404 error, even though other things like requesting signed URLs work with this endpoint URL. Hope this helps anyone.

Alex Ustinov
  • 1
0

Spent too much time on this. Here's a quick fix -

s3_client = boto3.client('s3')
s3_client.download_file('item1','item2', 'item3')

Here in .download_file

item1 = bucket name, e.g. 'lambda-ec2-test-bucket'

item2 = location of the key pair .pem file in that s3 bucket. e.g. 'keys/kp08092022.pem'

item3 = "tmp" folder in your lambda function where you want to save the downloaded file. e.g. '/tmp/keyname.pem'

Now the below code with the examples should work perfectly -

 s3_client = boto3.client('s3')

 #Download private key file from secure S3 bucket
 s3_client.download_file('lambda-ec2-test-bucket','kp08092022.pem', '/tmp/keyname.pem')
Gautam Dawar
  • 16
  • 2