Unit Testing by mocking S3 bucket-CodePudding

I am new to unit testing and I require to perform some simple unit tests for an object storage class.

I have a class named OSBucket as follows:

def __initBucket(self):        
        
        ecs_session = boto3.Session(
        aws_access_key_id="OSKEY",
        aws_secret_access_key="SECRETKEY"
        ) 

        OS_resource = ecs_session.resource('s3', verify=cert, endpoint_url=endpoint)
        self.mybucket = OS_resource.Bucket(OS_BUCKET)
                
        
def get_mybucket(self): 
        
   return self.mybucket


def download_file(self, fileName,filepath):
        
   self.mybucket.download_file(fileName, filepath)

def upload_file(self, filepath,dest_file_name):   
   self.mybucket.upload_file(filepath, '%s%s' % ("/",dest_file_name))

The method __initBucket is called in the constructor of the class.

How could I start creating a unit test class to test, for example, the download_file method?

CodePudding user response：

First Approach: using python mocks

You can mock the s3 bucket using standard python mocks and then check that you are calling the methods with the arguments you expect.

However, this approach won't actually guarantee that your implementation is correct since you won't be connecting to s3. For example, you can call non-existing boto functions if you misspelled their names - the mock won't throw any exception.

Second Approach: using moto

Moto is the best practice for testing boto. It simulates boto in your local machine, creating buckets locally so you can have complete end-to-end tests.

I used moto to write a couple of tests for your class (and I might have found a bug in your implementation - check the last test line - those are the arguments needed to make it find the file and not throw an exception)

import pathlib
from tempfile import NamedTemporaryFile

import boto3
import moto
import pytest
from botocore.exceptions import ClientError

from os_bucket import OSBucket


@pytest.fixture
def empty_bucket():
    moto_fake = moto.mock_s3()
    try:
        moto_fake.start()
        conn = boto3.resource('s3')
        conn.create_bucket(Bucket="OS_BUCKET")  # or the name of the bucket you use
        yield conn
    finally:
        moto_fake.stop()


def test_download_non_existing_path(empty_bucket):
    os_bucket = OSBucket()
    os_bucket.initBucket()
    with pytest.raises(ClientError) as e:
        os_bucket.download_file("bad_path", "bad_file")
    assert "Not Found" in str(e)


def test_upload_and_download(empty_bucket):
    os_bucket = OSBucket()
    os_bucket.initBucket()
    with NamedTemporaryFile() as tmp:
        tmp.write(b'Hi')
        file_name = pathlib.Path(tmp.name).name

        os_bucket.upload_file(tmp.name, file_name)
        os_bucket.download_file("/"   file_name, file_name)  # this might indicate a bug in the implementation

Final notes:

I actually gave a pycon lecture about this exact topic, watch it here
The source code for the lecture can be found here