Senario:

1. commit Athena query with boto3 and output to s3
2. download result in s3

Error: An error occurred (404) when calling the HeadObject operation: Not Found

It's weird that the file exists in S3 and I can copy it down with aws s3 cp command. But I just cannot download with boto3 and failed to execute head-object.

aws s3api head-object --bucket dsp-smaato-sink-prod --key /athena_query_results/c96bdc09-d545-4ee3-bc66-be3be928e3f2.csv

It does work. I've checked account policies and it has granted admin policy.

# snippets
def s3_donwload(url, target=None):
    # s3 = boto3.resource('s3')
    # client = s3.meta.client
    client = boto3.client("s3", region_name=constant.AWS_REGION, endpoint_url='https://s3.ap-southeast-1.amazonaws.com')
    
    s3_file = urlparse(url)

    if target:
        target = os.path.abspath(target)
    else:
        target = os.path.abspath(os.path.basename(s3_file.path))

    logger.info(f"download {url} to {target}...")
    client.download_file(s3_file.netloc, s3_file.path, target)
    logger.info(f"download {url} to {target} done!")

CodePudding user response：

Take a look at the value of s3_file.path -- does it start with a slash? If so, it needs to change because Amazon S3 keys do not start with a slash.

I suggest that you print the content of netloc, path and target to see what values it is actually passing.

It's a bit strange to use os.path with an S3 URL, so it might need some tweaking.