configuring Airflow ssh connection in google cloud composer

Question

I'm trying to configure a SSH connection from Airflow UI on google cloud composer environment to an on premise posgresql server Where I should store my private key ? How to pass to SSH connection config the private key location ?

Answer 1

First, you will need to add an SSH connection under:

 Airflow -> Admin -> Connections -> Connection Type (SSH)

That will allow you to use this connection in an operator to access the remote instance. Add your key to the Extra field (check key_file & host_key). Documentation here: https://airflow.apache.org/docs/apache-airflow-providers-ssh/stable/connections/ssh.html

Adding the file under the same GCS bucket having the dags will make it reachable by the Airflow workers. You can have a new directory under gads and name it keys if you want.

Then you will need to design your pipeline (dag) to be able to get your private key from the remote instance. You can use the SSHExecuteOperator or any other operator based on your design.

Check this question for more helpful details: Airflow: How to SSH and run BashOperator from a different server