Provide aws credentials to Airflow GreatExpectationsOperator

Question

I would like to use GreatExpectationsOperator to perform data quality validations.

The validation results data should be stored in S3. I don't see an option to send an airflow connection name to the GE operator, and the AWS credentials in my organization are stored in an airflow connection.

How can great expectations retrieve s3 credentials from airflow connection? and not from the default aws credentials in .aws dir?

Thanks!

Answer 1

We ended up creating a new oprator that inherit from GE operator and the operator get the connection as part of its ecxeute method.