Upload tar.gz file to S3 Bucket with Boto3 and Python-CodePudding

I am unable to load a tar.gz file from my local directory to an S3 bucket location. I've had no issues running the function below to upload any csv files but am getting the error: "Fileobj must implement read" error. I am using Boto3 and Python

The tar_file is the file on my local drive to upload to the S3 bucket location

import csv
import glob
import os
import tarfile
from datetime import date
from typing import Optional, Set
from io import BytesIO

import psycopg2
import boto3

from constants import (
    ARTIFACT_STORE,
    DB_HOST,
    DB_PASSWORD,
    DB_USER,
    EXCLUDED_TABLES,
    NIPR_DB_NAME,
    S3_ACCESS_KEY_ID,
    S3_SECRET_ACCESS_KEY,
    S3_ENDPOINT_URL,
    BUCKET_NAME
)

def upload_s3_file():
    tar_file = f"{ARTIFACT_STORE}/{date.today()}_cds.tar.gz"
    s3 = boto3.client('s3',endpoint_url=S3_ENDPOINT_URL,aws_access_key_id=S3_ACCESS_KEY_ID,aws_secret_access_key=S3_SECRET_ACCESS_KEY)
    with tarfile.open(tar_file,'r:gz') as tar:
        s3.upload_fileobj(tar,BUCKET_NAME,tar_file)

When I run the below function on a csv generated file to the S3 bucket, I have no issues:

s3 = boto3.client('s3',endpoint_url=S3_ENDPOINT_URL,aws_access_key_id=ACCESS_KEY,aws_secret_access_key=SECRET_KEY)
with open("test.csv", "rb") as f:
        s3.upload_fileobj(f,BUCKET_NAME, "test")

CodePudding user response：

The problem is you're supposed to pass a file object to upload_fileobj and not a tarfile object.

with open(tar_file,'rb') as tar:
    s3.upload_fileobj(tar,BUCKET_NAME,tar_file)