3

I'm trying to prepare a flow where we can regularly pull the available new files in third parties' on-prem server to our S3 using AWS Transfer family. I read this documentation https://aws.amazon.com/blogs/storage/how-discover-financial-secures-file-transfers-with-aws-transfer-family/, but it was not clear on setting up and configuring the process. Can someone share any clear documentation or reference links on using AWS Transfer Family to pull files from external on-prem server to our S3?

Sampath
  • 153
  • 2
  • 13
  • Can you provide detail on where you are stuck on this? Some information on what has been tried and if any error. – Nagaraj Tantri Feb 04 '22 at 13:35
  • I created a AWS Transfer for SFTP server and connected that server with my IAM role which have access to a S3 bucket. I want to understand how to connect my SFTP server and external server so that I can pull files directly from the external server to my S3 bucket using AWS Transfer family. – Sampath Feb 07 '22 at 09:56
  • For connecting from External Server to SFTP Server, you can use a simple SFTP Client on your external server and then dump it to your SFTP Server. Check this https://linuxize.com/post/how-to-use-linux-sftp-command-to-transfer-files/ . If you have configured your AWS Transfer file properly, then this should just push the data files from your SFTP To S3 – Nagaraj Tantri Feb 07 '22 at 11:17

2 Answers2

1

@Sampath, I think you misunderstood the available features of the AWS Transfer service. That service is actually acting as a serverless SFTP with AWS S3 as the backend storage to which you can connect via SFTP protocol (now supports FTP and FTPS as well). You can either PUSH data to S3 or PULL data from S3 via AWS Transfer service. You cannot PULL data into S3 from anywhere else via AWS Transfer service alone.

You may have to use any other solution like a Python Script running on AWS EC2 for that purpose.

Another solution would be to connect the external third-party server to the AWS Transfer Service and that server PUSHES files on S3 via AWS Transfer.

As per your use case, I think you need a simple solution that connects to an external third-party server and copies files from it to the AWS S3 bucket. It can be done via a Python script as well and you can run it on either AWS EC2, AWS ECS, AWS Lambda, AWS Batch, etc, depending on the specifications and requirements.

I have used AWS Transfer once I found it to be very expensive and went on with AWS EC2 instead. In the case of AWS EC2, you can even buy reserved instances to further reduce the cost. If the task is just about copying files from an external server to S3 and the copy job will never take more than 10 minutes, then it is better to run it on AWS Lambda.

In short, you cannot PULL data from any server into S3 using the AWS Transfer service. You can only PUSH data to or PULL data from S3 using the AWS Transfer service.

References to some informative blogs:

Abdullah Khawer
  • 4,461
  • 4
  • 29
  • 66
  • I already read this documentation, this links contains info about creating a cloudformation template and using Fargate task, and then providing SFTP access to users using AWS transfer family. My use case is different - I want to pull data from external third party server using AWS Transfer Family. Thank you. – Sampath Feb 07 '22 at 09:53
  • @Sampath, I have updated my answer, check now. – Abdullah Khawer Feb 07 '22 at 11:59
  • You said - "Another solution would be to connect the external third-party server to the AWS Transfer Service and that server PUSHES files on S3 via AWS Transfer." Can I know how to connect third-party server to our AWS Transfer server? – Sampath Feb 07 '22 at 13:20
  • @Sampath by creating a user on AWS Transfer Service by specifying the public key created on the external third-party server. Just like we do for SSH access. See this: https://docs.aws.amazon.com/transfer/latest/userguide/getting-started.html#getting-started-user – Abdullah Khawer Feb 07 '22 at 15:12
  • 1
    Thanks for your answer and the clarification. This is helpful. – Sampath Feb 14 '22 at 05:01
  • @Sampath My pleasure. :) – Abdullah Khawer Feb 14 '22 at 13:42
1

With the AWS Transfer Family service you can create servers that uses SFTP, FTPS, and FTP protocols for your file transfers, and use the Amazon S3 and EFS as domains to store and access your files.

To connect your on-premise servers with the Transfer Family server you will need to use a service like File Gateway/Storage Gateway and connect via HTTPS to S3 to sync your files.

Your architecture will be something like this:

enter image description here

If you want more details of how to connect with your on-premises servers with the AWS S3/Transfer Family services take a look on this blog post: Centralize data access using AWS Transfer Family and AWS Storage Gateway

valdeci
  • 13,962
  • 6
  • 55
  • 80