Using Apache Airflow Tool, Implement a DAG for a batch processing pipeline to get a directory from a remote system

Question

Using Apache airflow tool, how can I implement a DAG for the following Python code. The task accomplished in the code is to get a directory from GPU server to local system. Code is working fine in Jupyter notebook. Please help to implement in Airflow...I'm very new to this. Thanks.

import pysftp
import os
myHostname = "hostname"
myUsername = "username"
myPassword = "pwd"

with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword) as sftp:
    print("Connection successfully stablished ... ")
    src = '/path/src/'
    dst = '/home/path/path/destination'
    os.mkdir(dst)
    sftp.get_d(src, dst, preserve_mtime=True)
    print("Fetched source images from GPU server to local directory")
# connection closed automatically at the end of the with-block```

score 1 · Accepted Answer · answered Aug 13 '20 at 21:30

For SFTP duties, Airflow provides SFTOperator that you can use directly.
Alternatively it's corresponding SFTPHook can be used with a simple PythonOperator

I acknowledge there aren't many examples, but this might be helpful
For SSH-connection, see this

Using Apache Airflow Tool, Implement a DAG for a batch processing pipeline to get a directory from a remote system

1 Answers1