Home > Software engineering >  Reduce verbosity/frequency of Amazon S3 AWS upload ProgressListener?
Reduce verbosity/frequency of Amazon S3 AWS upload ProgressListener?

Time:11-09

I am doing a multi-part file upload to S3. I added a ProgressListener to print out progress and it works great. Too great. The file I am uploading is several GB large and the messages I am getting logged are way too granular. I am getting several thousand of these:

2021-11-04 06:56:07 INFO  GenericSftpLoader:190 - transfered bytes: 8192

Is there any way to reduce this from printing out every 8192 bytes and instead something more reasonable like 1048576 bytes?

My actual code is in Scala but I followed this example closely from the AWS SDK documentation:

TransferManager tm = new TransferManager(new ProfileCredentialsProvider());        

    PutObjectRequest request = new PutObjectRequest(
            existingBucketName, keyName, new File(filePath));
    
    request.setGeneralProgressListener(new ProgressListener() {
        @Override
        public void progressChanged(ProgressEvent progressEvent) {
            System.out.println("Transferred bytes: "   
                    progressEvent.getBytesTransferred());
        }
    });       

    Upload upload = tm.upload(request);

CodePudding user response:

So I was able to get a solution. Here's what I did:

TransferManager tm = new TransferManager(new ProfileCredentialsProvider());

PutObjectRequest request = new PutObjectRequest(
        existingBucketName, keyName, new File(filePath));

request.setGeneralProgressListener(new ProgressListener() {
    private Long myBytes = 0
    @Override
    public void progressChanged(ProgressEvent progressEvent) {
        myBytes  = progressEvent.getBytesTransferred();
        if (bytesTransferred > 1048576) {
            System.out.println("Transferred bytes: "   
                    myBytes);
            myBytes = 0;
        }
    }
});       

Upload upload = tm.upload(request);
  • Related