53

I'm downloading files from AWS S3 Bucket like so:

import boto3

s3client = boto3.client("s3")
s3 = boto3.resource('s3')

bucket_name = 'practice_bucket'
bucket = s3.Bucket(bucket_name)

for obj in bucket.objects.all():
    filename = obj.key.rsplit('/')[-1]
    s3client.download_file(bucket_name, obj.key, "/txt/" + filename)

When attempting to place some files under a subdirectory, e.g. /txt/, I get the error:

botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

Oddly, it works on other file types using the same method, but doesn’t work for ones with .json extension.

What could be the issue? I even tried without placing them in an absolute subdirectory path, and I get no error and downloads the file onto the same directory as the script downloading. But when I actually define the path to download the file to, I get the error.

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
Dan Me
  • 2,143
  • 4
  • 19
  • 19
  • 18
    I just tried your code and it worked fine. If you are getting the 404 error, that means there is some mismatch between the requested key and the actual key. – 2ps Jul 04 '17 at 00:45
  • 16
    This question should be reopened. I have the same problem. He provided the desired behavior, the specific problem, and the error, as well as code. I am goping to have to go make another ticket for the exact same problem now. My key matches. I logged it to make sure. – Christopher Pisz Oct 02 '19 at 23:28
  • 4
    Me too @ChristopherPisz. I have the same problem. Perhaps that is the AWS issue? I have been using my code in 5 months and it works well until today, I got that error. I actually don't know why? – Han Van Pham Oct 03 '19 at 08:28
  • 5
    Seems related to permissions in my case. I remade the lambda permission policies and it started working again. – Christopher Pisz Oct 03 '19 at 16:56
  • 1
    Check this link out for `404` [Accessing a Bucket](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/migrations3.html?#accessing-a-bucket) Hope it helps – Santhosh Dhaipule Chandrakanth Dec 05 '19 at 07:52
  • 1
    I ran into this issue when passing a folder as the second argument to download_file. Programmer mistake, but also a rather cryptic error – Will Ayd May 06 '20 at 16:32
  • 6
    Happened to me too. For me it works when the key does not start as an absolute path, ie it does not start with "/". – JJCV Dec 11 '20 at 09:40
  • 1
    I get this error on production side, in dev side I do not get it, however both codes are identical... both run in windows, I cannot figure out what the difference is... – Flying Turtle Mar 23 '21 at 08:34
  • 1
    @JJCV : Thank you for your valuable comment. I just used the urllib.parse's urlparse and it puts the `/` in the front of the key. The error should be different to be honest. – user238607 Apr 09 '21 at 20:06
  • 4
    I got this error when I accidentally misspelled the file name for the file I was downloading. – NorahKSakal May 01 '21 at 04:33
  • 1
    I encounter same issues even I passed correct parameters `api_params = {'Bucket': 'file-exporter', 'Key': 'e_group.json'}`. Just, difference is I am using async approach. – Suyog Shimpi Sep 27 '21 at 12:04
  • 1
    For me the answer was that the path to my s3 container's volume was wrong, so the file I was trying to open didn't exist. Another reopen vote here -- not just a comment, I actually voted. – Noumenon Oct 06 '21 at 19:36
  • If anyone is getting this on DigitalOcean App Platform comment here with your findings. This is such a strange error and I can't entirely pinpoint what is causing it. Sometimes it randomly appears and then is ok again for a new deploys. It makes no sense. When I "collectstatic" from dev/local to cloud (AWS) so that all static files are already in place before the real deploy, and then do a "Force Rebuild" with "clear cache" on the app it seems to work fine. Otherwise it appears to just randomly break. The error message itself really doesn't help either. I'll keep digging. – sebieire Apr 05 '22 at 15:36

2 Answers2

6

Your code is correct.

botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

This error is thrown when the object you are trying to fetch is not present in the bucket.

Vikas Zingade
  • 134
  • 1
  • 2
  • 5
  • 2
    Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Neeraj Dec 07 '21 at 11:40
  • This answer was perfect as is and provided me with all the information I needed to solve my problem. In my case, I just added a try/except botocore.exceptions.ClientError around the download call which results in catching the error when the expected file doesn't exist. – Jeremy Goodell Sep 28 '22 at 20:25
0

Upgrading the aiobotocore solved the issue for me.

pip install --upgrade aiobotocore
iselim
  • 309
  • 3
  • 8