Env file:
AWS_ACCESS_KEY_ID=AAAAAAAAAAAAAAAAAA
AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxx
AWS_DEFAULT_REGION=eu-west-1
AWS_DEFAULT_OUTPUT=json
Config file:
[profile a]
credential_source = Environment
region=eu-west-1
role_arn=arn
[profile b]
credential_source = Environment
region=eu-west-1
role_arn=arn
[profile c]
credential_source = Environment
region=eu-west-1
role_arn=arn
Dockerfile content:
FROM apache/airflow:2.1.2-python3.8
ARG AIRFLOW_USER_HOME=/opt/airflow
ENV PYTHONPATH "${PYTHONPATH}:/"
RUN pip install -r requirements.pip
ADD ./my_path/aws/config /home/airflow/.aws/config
Volume in compose:
- ./my_path/aws:/home/airflow/.aws
If I run:
docker exec -it my-container bash
aws configure list
I see:
Name Value Type Location
---- ----- ---- --------
profile <not set> None None
access_key *******************A env
secret_key *******************x env
region eu-west-1 env AWS_DEFAULT_REGION
The error I get when running some Python3 code using boto3 is:
Error when retrieving credentials from Environment: No credentials found in credential_source referenced in profile.
Why is happening that if acces_key and secret are setup, and config file is indication that source is environment?
Extra tests:
I run: aws s3 ls inside the container and it list everything, so credentials are correctly setup from the env file.
Moreover the python code raising the issue is:
class Boto3AwsClient(object):
def __init__(self, localhost, profile):
self.running_localhost = localhost
self.profile = profile
if self.running_localhost == 1:
self.session = boto3.Session(profile_name=self.profile)
else:
self.session = boto3.Session()
def aws_client_connect(self, service=None):
if service is None:
raise ValueError('Service is not defined in new boto3 session.')
service_client = self.session.client(service)
return service_client
I tried also to settup the config file like this:
[default]
credential_source = Environment
region=eu-west-1
output=json
[profile a]
region=eu-west-1
role_arn=arn
[profile b]
region=eu-west-1
role_arn=arn
[profile c]
region=eu-west-1
role_arn=arn
And do not work neither. The only feasible way is to have a config file and the credentials file, and do not setup the ENV variables. The problem with that is it's not a good practice.
