Copy of file from S3 to SFTP server using python-CodePudding

I want to copy a file from AWS S3 to SFTP server without downloading to local using python,any help on this can be appreciated, thank you.

Sample python code to run.

CodePudding user response：

There is no command in Amazon S3 to 'send' a file to a destination. It is only possible to 'download' (or 'read') the contents of a file.

Therefore, the only way to accomplish your goal is to download the file from S3, then upload to the SFTP server.

You could take advantage of the features of smart-open · PyPI, which lets you 'open' a file on S3 or SFTP as if it were a local file, but please note this would still involve 'downloading' and 'uploading' in the contents of the file.

CodePudding user response：

You will need to download the file from S3, then upload it to the SFTP server. That said, you do not need to download the file to your local machine's drive, you can transfer down the data and immediately send it along to S3.

So, you can use boto3 to download the file, then use paramiko to send the file back up to the SFTP server. Since boto3 can return a file object, and paramiko can use a file object to upload, it's mostly a matter of creating the connections and putting them together.

#!/usr/bin/env python3

import paramiko
import boto3

# Example settings for a S3 bucket/key
AWS_S3_BUCKET = "example-bucket"
AWS_S3_KEY = "example/existing_object.dat"

# Example settings for a SFTP server
SSH_HOST = "ssh.hostname.example.com"
SSH_USERNAME = "username"
SSH_PRIVATE_KEY = "/path/to/ssh_key.pem"
SSH_FILENAME = "/path/to/target/filename"

# Connect to S3 and start transferring the file
s3 = boto3.client('s3')
s3_object = s3.get_object(Bucket=AWS_S3_BUCKET, Key=AWS_S3_KEY)

# Build the SSH/SFTP connection:
pkey = paramiko.RSAKey.from_private_key_file(SSH_PRIVATE_KEY)
transport = paramiko.Transport((SSH_HOST, 22))
transport.connect(username=SSH_USERNAME, pkey=pkey)
sftp = paramiko.SFTP.from_transport(transport)

# Connections are done, use putfo to transfer the data from S3
# to the SFTP server
print("Transferring file from S3 to SFTP...")
sftp.putfo(s3_object['Body'], SSH_FILENAME)

# All done, clean up
sftp.close()
print("Done")