1

I run airflow in docker and has volumes setting as

- ~/.aws:/usr/local/airflow/.aws:rw

I can see my local ~/.aws/credentials has:

[some_profile]
aws_access_key_id = xxxxx
aws_secret_access_key = xxxxx

and I can see the same content in airflow worker /usr/local/airflow/.aws/credentials, and aws configure list can return expected results like:

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                 datalake              env    ['xxx_PROFILE', 'AWS_PROFILE']
access_key     ****************xxxx shared-credentials-file
secret_key     ****************xxxx shared-credentials-file
    region                <not set>             None    None

and cmd like aws s3 ls works locally. However for

sts_client = boto3.client('sts')
assumed_role = sts_client.assume_role(RoleArn=role_arn, RoleSessionName='xxx')

it throws:

botocore.exceptions.NoCredentialsError: Unable to locate credentials

What am I missing here? Thanks.

processadd
  • 101
  • 3
  • 5

1 Answers1

0

The problem could be here. When you're declaring [some_profile] for the credentials file, it should be like this:

[default]
aws_access_key_id = xxxxx
aws_secret_access_key = xxxxx

There must be a default profile in your credentials.

If that doesn't work, you can always set the credentials manually

First, set the profile

[some_profile]
aws_access_key_id = xxxxx
aws_secret_access_key = xxxxx

Then, your Python code should look like this

os.environ['AWS_PROFILE'] = "some_profile"
os.environ['AWS_DEFAULT_REGION'] = "some_region"

s3 = boto3.client('s3', region_name='some_region')
strider
  • 683
  • 9
  • 25
  • Thanks @strider. Seems the container worker looks up `default` profile. If I set ``` os.environ['AWS_PROFILE'] = "some_profile" os.environ['AWS_DEFAULT_REGION'] = "some_region" ``` It can look up some_profile. Is there a way not to change the Python but to use the profile I specified? I tried AWS_PROFILE=some_profile or put profile in docker compose but not work. – processadd Jan 07 '23 at 22:58