3

I'm trying to query postgres from an MWAA instance of airflow. I'm not sure if there is a conflict due to airflow itself having a different version of postgres for its metadata or what, but I get this error when connecting to postgres:

  File "/usr/local/airflow/dags/transactions/transactions.py", line 62, in load_ss_exposures_to_s3
    ss_conn = psycopg2.connect(
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: SCRAM authentication requires libpq version 10 or above

Locally I have psycopg2 version 2.9.5 and libpq version 140005. MWAA is using psycopg2 2.9.5 and libpq 90224. Is there a way for me to force MWAA to use another version? Maybe through airflow plugins? Airflow version is 2.4.3.

RagePwn
  • 411
  • 2
  • 8
  • 22

2 Answers2

2

In case anyone else encounters this issue when upgrading to MWWA Airflow 2.4.3, I managed to resolve this issue by adding psycopg2-binary to the requirements.txt file. I didn't specify a version.

Alex
  • 21,273
  • 10
  • 61
  • 73
  • Thanks! this would have been very hard to figure out. For other people coming here, I had to add this dependency to the `requirement.txt` as this answer says, instead of the `requirements` of my `PythonVirtualenvOperator` (couldn't make it work with that one) – Daniel Jul 24 '23 at 19:45
-1

To update the libs version, in AWS MWAA, you can add a requirements file to S3, and configure MWAA to install it by selecting the file and updating the environment (doc).

For your problem, you can try installing the same version you have in localhost, but if you are using graviton VM in MWAA (arm64), it can be another problem, like installing psycopg2-binary which is not compatible with arm64, and in this case you need to install psycopg2:

pip install psycopg2
Hussein Awala
  • 4,285
  • 2
  • 9
  • 23
  • They note they're using psycopg2==2.9.5 in both local and MWAA. You can't control the amazonlinux package versions with a requirements file. – holly.evans May 05 '23 at 19:54