0

I'd like to get all archives from a specific directory on S3 bucket like the following:

def get_files_from_s3(bucket_name, s3_prefix):
    files = []
    s3_resource = boto3.resource("s3")
    bucket = s3_resource.Bucket(bucket_name)
    response = bucket.objects.filter(Prefix=s3_prefix)
    for obj in response:
        if obj.key.endswidth('.zip'):
            # get all archives
            files.append(obj.key)
    return files

My question is about testing it; because I'd like to mock the list of objects in the response to be able to iterate on it. Here is what I tried:

from unittest.mock import patch
from dataclasses import dataclass

@dataclass
class MockZip:
    key = 'file.zip'

@patch('module.boto3')
def test_get_files_from_s3(self, mock_boto3):
    bucket = mock_boto3.resource('s3').Bucket(self.bucket_name)
    response = bucket.objects.filter(Prefix=S3_PREFIX)
    response.return_value = [MockZip()]
    files = module.get_files_from_s3(BUCKET_NAME, S3_PREFIX)
    self.assertEqual(['file.zip'], files)

I get an assertion error like this: E AssertionError: ['file.zip'] != []

Does anyone have a better approach? I used struct but I don't think this is the problem, I guess I get an empty list because the response is not iterable. So how can I mock it to be a list of mock objects instead of just a MockMagick type?

Thanks

ezzeddin
  • 499
  • 2
  • 5
  • 25

1 Answers1

1

You could use moto, which is an open-source libray specifically build to mock boto3-calls. It allows you to work directly with boto3, without having to worry about setting up mocks manually.

The testfunction that you're currently using would look like this:

from moto import mock_s3

@pytest.fixture(scope='function')
def aws_credentials():
    """Mocked AWS Credentials, to ensure we're not touching AWS directly"""
    os.environ['AWS_ACCESS_KEY_ID'] = 'testing'
    os.environ['AWS_SECRET_ACCESS_KEY'] = 'testing'
    os.environ['AWS_SECURITY_TOKEN'] = 'testing'
    os.environ['AWS_SESSION_TOKEN'] = 'testing'


@mock_s3
def test_get_files_from_s3(self, aws_credentials):
    s3 = boto3.resource('s3')
    bucket = s3.Bucket(self.bucket_name)
    # Create the bucket first, as we're interacting with an empty mocked 'AWS account'
    bucket.create()

    # Create some example files that are representative of what the S3 bucket would look like in production
    client = boto3.client('s3', region_name='us-east-1')
    client.put_object(Bucket=self.bucket_name, Key="file.zip", Body="...")
    client.put_object(Bucket=self.bucket_name, Key="file.nonzip", Body="...")

    # Retrieve the files again using whatever logic
    files = module.get_files_from_s3(BUCKET_NAME, S3_PREFIX)
    self.assertEqual(['file.zip'], files)

Full documentation for Moto can be found here: http://docs.getmoto.org/en/latest/index.html

Disclaimer: I am a maintainer for Moto.

Bert Blommers
  • 1,788
  • 2
  • 13
  • 19
  • Good to know that you put some PRs in moto. Thanks for the answer but unfortunately, it's giving me the same `assertionError`. Is there something else I'm doing wrong? and does this decorator `@pytest.fixture(scope='function')` do? – ezzeddin Dec 05 '21 at 11:49
  • awesome! it's working now! I was missing the full path to the file on S3 which should have included the s3 prefix. So here is a modification to the put_object line: `client.put_object(Bucket=self.bucket_name, Key=self.s3_prefix + "file.zip", Body="...")` similar edit for the line after – ezzeddin Dec 05 '21 at 12:59