Background: I have an AWS EC2 instance in which I have my analytical Python scripts which output files to the AWS storage.
I have been using the command terminal from my local machine to upload the input data and download those outputs. But this has been requiring me to manually do this step and I wanted to know if there was a methodology of having code in the script pipeline that would push my outputs to my local directory automatically? Right now I am more concerned with downloading the output rather than uploading the input.
To this point, I have been using the PSCP calls to move the data back and forth with the following commands from the terminal:
pscp -i <ppk-file-local-dir> <AWS file-dir> <local dir>
I want to be able to take this information and write it within the script itself, so I can stop writing out the command every time I need to download the data. I am seeing some potential with using the subprocess package, but I just don't have a good grasp of it yet.
