Download file from AWS S3 using Python

Question

I am trying to download a file from Amazon S3 bucket to my local using the below code but I get an error saying "Unable to locate credentials"

Given below is the code I have written:

from boto3.session import Session
import boto3

ACCESS_KEY = 'ABC'
SECRET_KEY = 'XYZ'

session = Session(aws_access_key_id=ACCESS_KEY,
              aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
your_bucket = s3.Bucket('bucket_name')

for s3_file in your_bucket.objects.all():
    print(s3_file.key) # prints the contents of bucket

s3 = boto3.client ('s3')

s3.download_file('your_bucket','k.png','/Users/username/Desktop/k.png')

Could anyone help me on this?

score 80 · Accepted Answer · answered Apr 30 '18 at 15:37

80

You are not using the session you created to download the file, you're using s3 client you created. If you want to use the client you need to specify credentials.

your_bucket.download_file('k.png', '/Users/username/Desktop/k.png')

or

s3 = boto3.client('s3', aws_access_key_id=... , aws_secret_access_key=...)
s3.download_file('your_bucket','k.png','/Users/username/Desktop/k.png')

answered Apr 30 '18 at 15:37

Joaquín Bucca

1,060
8
11

7

@Joaquin please add documentation to where you found this for a more complete answer. Nevertheless +1 – PolarBear10 Apr 01 '19 at 14:57

Increasingly Idiotic · Answer 2 · 2021-06-02T19:45:32.263

From an example in the official documentation, the correct format is:

import boto3

s3 = boto3.client('s3', aws_access_key_id=... , aws_secret_access_key=...)
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME')

You can also use a file-like object opened in binary mode.

s3 = boto3.client('s3', aws_access_key_id=... , aws_secret_access_key=...)
with open('FILE_NAME', 'wb') as f:
    s3.download_fileobj('BUCKET_NAME', 'OBJECT_NAME', f)
    f.seek(0)

The code in question uses s3 = boto3.client ('s3'), which does not provide any credentials.

The format for authenticating a client is shown here:

import boto3
client = boto3.client(
    's3',
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
    aws_session_token=SESSION_TOKEN,
)

# Or via the Session
session = boto3.Session(
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
    aws_session_token=SESSION_TOKEN,
)

And lastly you can also re-use the authenticated session you created to get the bucket, and then download then file from the bucket.

from boto3.session import Session
import boto3

ACCESS_KEY = 'ABC'
SECRET_KEY = 'XYZ'

session = Session(aws_access_key_id=ACCESS_KEY,
              aws_secret_access_key=SECRET_KEY)

# session is authenticated and can access the resource in question 
session.resource('s3')
    .Bucket('bucket_name')
    .download_file('k.png','/Users/username/Desktop/k.png')

For your use of `download_fileobj` may I suggest you add the line: `f.seek(0)`. The file is left in an non-deterministic state. This line ensures you start reading it back from the beginning (just spent some time figuring this out myself!) — Robert King, Nov 12 '19 at 11:35
@RobertKing do you mind elaborating on this point? Why add `f.seek(0)`? — ramhiser, Aug 24 '21 at 20:50

score 2 · Answer 3 · answered Feb 11 '19 at 01:14

For others trying to download files from AWS S3 looking for a more user-friendly solution with other industrial-strength features, check out https://github.com/d6t/d6tpipe. It abstracts the S3 functions into a simpler interface. It also supports directory sync, uploading files, permissions and many other things you need to sync files from S3 (and ftp).

import d6tpipe
api = d6tpipe.api.APILocal() # keep permissions locally for security

settings = \
{
    'name': 'my-files',
    'protocol': 's3',
    'location': 'bucket-name',
    'readCredentials' : {
        'aws_access_key_id': 'AAA',
        'aws_secret_access_key': 'BBB'
    }
}

d6tpipe.api.create_pipe_with_remote(api, settings)

pipe = d6tpipe.Pipe(api, 'my-files')
pipe.scan_remote() # show all files
pipe.pull_preview() # preview
pipe.pull(['k.png']) # download single file
pipe.pull() # download all files

pipe.files() # show files
file=open(pipe.dirpath/'k.png') # access file

score 1 · Answer 4 · answered Mar 09 '21 at 05:30

You can setup your AWS profile with awscli to avoid introduce your credentials in the file. First add your profile:

aws configure --profile account1

Then in your code add:

aws_session = boto3.Session(profile_name="account1")
s3_client = aws_session.client('s3')

score 0 · Answer 5 · answered Jan 03 '23 at 05:46

FileName:

can be any name; with that name; file will be downloaded.
It can be added to any existing local directory.

Key:

Is the S3 file path along with the file name in the end.
It does not start with a backslash.

Session()

It automatically picks the credentials from ~/.aws/config OR ~/.aws/credentials

If not you need to explicitly pass that.

from boto3.session import Session
import boto3

# Let's use Amazon S3
s3 = boto3.resource("s3")

# Print out bucket names to check you have accessibility
# for bucket in s3.buckets.all():
#     print(bucket.name)

session = Session()
OR
session = Session(aws_access_key_id="AKIAYJN2LNOU",
                  aws_secret_access_key="wMyT0SxEOsoeiHYVO3v9Gc",
                  region_name="eu-west-1")    

session.resource('s3').Bucket('bucket-logs').download_file(Key="logs/20221122_0_5ee03da676ac566336e2279decfc77b3.gz", Filename="/tmp/Local_file_name.gz")

Download file from AWS S3 using Python

5 Answers5

FileName:

Key:

Session()

Linked