1

I am facing something similar to How to load file from custom hosted Minio s3 bucket into pandas using s3 URL format?

however, I already have an initialized s3 session (from boto3). How can I get the credentials returned from it to feed these directly to pandas? I.e. how can I extract the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the initialized boto3 s3 client?

Georg Heiler
  • 16,916
  • 36
  • 162
  • 292
  • aren't you the one that is initialising the client with the creds? initialise pandas in the same way? can you show some code please? – Ermiya Eskandary May 15 '22 at 14:23

1 Answers1

6

You can use session.get_credentials

import boto3

session = boto3.Session()
credentials = session.get_credentials()

AWS_ACCESS_KEY_ID = credentials.access_key
AWS_SECRET_ACCESS_KEY = credentials.secret_key
AWS_SESSION_TOKEN = credentials.token

If you only have access to boto client (like the S3 client), you can find the credentials hidden here:

client = boto3.client("s3")

client._request_signer._credentials.access_key
client._request_signer._credentials.secret_key
client._request_signer._credentials.token

If you don't want to handle credentials (I assume you're using the SSO here), you can load the S3 object directly with pandas: pd.read_csv(s3_client.get_object(Bucket='Bucket', Key ='FileName').get('Body'))

RobinFrcd
  • 4,439
  • 4
  • 25
  • 49
  • Thanks. What you are suggesting is an exciting option. But for now, I think I want to keep using the path (with the extra options) as this is more generic. However, `session.meta.client` is the object I am dealing with (and not `session`). How would I need to modify the parameter accordingly for this type? – Georg Heiler May 15 '22 at 14:48
  • 1
    I edited my answer with an example with a S3 client, hope this one helps – RobinFrcd May 15 '22 at 15:14