trying to use boto copy to s3 unless file exists-CodePudding

in my code below, fn2 is the local file and "my_bucket_object.key" is a list of files in my s3 bucket.

I am looking at my local files, taking the latest one by creation date and then looking at the bucket and I only want to copy the latest one there (this is working) but not if it exists already. What is happening is that, even if the file is there in the bucket, the latest file is still getting copied, overwriting the one in the bucket with the same name. the filename of the latest file is "bats6.csv" I figured that specifying 'in' and 'not in' conditions, this would ensure that the file did not get copied if one with the same name is already there, but this isnt working. Here is the code. Thanks alot.

 import boto3
 import botocore
 import glob, os
 import datetime
 import os

 exchanges = ["bats"]

 for ex in exchanges:
     csv_file_list = glob.glob(f"H:\SQL_EXPORTS\eod\candles\{ex}\\*.csv")
     latest_file = max(csv_file_list, key=os.path.getctime)

     path = f'H:\\SQL_EXPORTS\\eod\\candles\\{ex}\\'
     fileBaseName = os.path.basename(latest_file).split('.')[0]

     fn = path   fileBaseName   ".csv"
     fn2 = fileBaseName   ".csv"

     print(fn2)

     s3 = boto3.resource('s3')
     my_bucket = s3.Bucket(f'eod-candles-{ex}')

     for my_bucket.object in my_bucket.objects.all():
         print(my_bucket.object.key)
         if fn2 not in my_bucket.object.key:
             #s3.meta.client.upload_file(fn, my_bucket, fn2)
             s3.meta.client.upload_file(fn, f'eod-candles-{ex}', fn2)
         elif fn2 in my_bucket.object.key:
             print("file already exists")

CodePudding user response：

You could make a List of the object keys and then check whether it exists:

object_keys = [object.key for object in my_bucket.objects.all()]
if fn2 not in object_keys:
    s3.meta.client.upload_file(fn, f'eod-candles-{ex}', fn2)