Regularly pull files from On-Prem server to S3 using AWS Transfer family-CodePudding

The

If you want more details of how to connect with your on-premises servers with the AWS S3/Transfer Family services take a look on this blog post: Centralize data access using AWS Transfer Family and AWS Storage Gateway

CodePudding user response：

@Sampath, I think you misunderstood the available features of the AWS Transfer service. That service is actually acting as a serverless SFTP with AWS S3 as the backend storage to which you can connect via SFTP protocol (now supports FTP and FTPS as well). You can either PUSH data to S3 or PULL data from S3 via AWS Transfer service. You cannot PULL data into S3 from anywhere else via AWS Transfer service alone.

You may have to use any other solution like a Python Script running on AWS EC2 for that purpose.

Another solution would be to connect the external third-party server to the AWS Transfer Service and that server PUSHES files on S3 via AWS Transfer.

As per your use case, I think you need a simple solution that connects to an external third-party server and copies files from it to the AWS S3 bucket. It can be done via a Python script as well and you can run it on either AWS EC2, AWS ECS, AWS Lambda, AWS Batch, etc, depending on the specifications and requirements.

I have used AWS Transfer once I found it to be very expensive and went on with AWS EC2 instead. In the case of AWS EC2, you can even buy reserved instances to further reduce the cost. If the task is just about copying files from an external server to S3 and the copy job will never take more than 10 minutes, then it is better to run it on AWS Lambda.

In short, you cannot PULL data from any server into S3 using the AWS Transfer service. You can only PUSH data to or PULL data from S3 using the AWS Transfer service.

References to some informative blogs:

How Discover Financial secures file transfers with AWS Transfer Family

Moving external site data to AWS for file transfers with AWS Transfer Family

Easy SFTP Setup with AWS Transfer Family