Running coverage.py on project with moto causes jinja2 error

Question

I am mocking AWS calls to s3 with the @mock_s3 decorator from moto and boto3. I am running tests with "coverage run" from my root directory.

Whenever my tests involve any sort of command involving listing the contents of AWS S3, I am getting this error:

  File "/Users/Name/Documents/project-name/test/lib/python3.7/site-packages/jinja2/compiler.py", line 1044, in visit_For
    if node.test:
AttributeError: 'For' object has no attribute 'test'

Some example of commands that are failing with this error are:

s3_client = s3_session.client("s3")
s3_client.list_buckets() #fails at this line

and

s3_session = boto3.session.Session()
s3_client = s3_session.client("s3")
paginator = s3_client.get_paginator("list_objects_v2")
iterator = paginator.paginate(Bucket="mybucket", Prefix="mypath")
keys = iterator.search(query)
regex = re.compile('(\S)')
data_files = list(filter(regex.search, keys)) #fails at this line

Before running these lines of code, I am using boto3 to create buckets and objects within these buckets.

This is an example of a test:

@mock_s3
class TestModel(TestCase):

    def test_my_model_save(self):

        s3_client = boto3.client('s3', region_name='us-east-1') 
        s3_client.create_bucket(Bucket='mybucket') # printing the response from this returns {'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {}, 'RetryAttempts': 0}}
        s3_client.put_object(Bucket='mybucket', Key="mypath", Body="details") # printing the response from this returns {'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'etag': '"27792947ed5d5da7c0d1f43327ed9dab"', 'last-modified': 'Fri, 15 Mar 2019 20:15:44 GMT', 'content-length': '7'}, 'RetryAttempts': 0}, 'ETag': '"27792947ed5d5da7c0d1f43327ed9dab"'}
        print(s3_client.list_buckets()) #code breaks here

I recognize that this test is not complete (missing assertions and not actually testing my code), this is just a generic case to demonstrate the error.

I am also noticing that the response from create_bucket and put_object does not seem to match the boto3 documentation for these two functions (see comments in above sample test code).

Here is the full traceback when executing my test:

    Traceback (most recent call last):
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/moto/core/models.py", line 74, in wrapper
    result = func(*args, **kwargs)
  File "/Users/user_name/Documents/project_name/api/tests/test_my_Model.py", line 24, in test_my_model_save
    print(s3_client.list_buckets())
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/client.py", line 611, in _make_api_call
    operation_model, request_dict)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/endpoint.py", line 136, in _send_request
    success_response, exception):
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/endpoint.py", line 210, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/endpoint.py", line 176, in _get_response
    responses = self._event_emitter.emit(event_name, request=request)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/moto/core/models.py", line 292, in __call__
    status, headers, body = response_callback(request, request.url, request.headers)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/moto/s3/responses.py", line 116, in bucket_response
    response = self._bucket_response(request, full_url, headers)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/moto/s3/responses.py", line 138, in _bucket_response
    return self.all_buckets()
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/moto/s3/responses.py", line 53, in all_buckets
    template = self.response_template(S3_ALL_BUCKETS)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/moto/core/responses.py", line 103, in response_template
    return self.environment.get_template(template_id)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/jinja2/environment.py", line 830, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/jinja2/environment.py", line 804, in _load_template
    template = self.loader.load(self, name, globals)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/jinja2/loaders.py", line 125, in load
    code = environment.compile(source, name, filename)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/jinja2/environment.py", line 581, in compile
    defer_init=defer_init)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/jinja2/environment.py", line 543, in _generate
    optimized=self.optimized)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/jinja2/compiler.py", line 82, in generate
    generator.visit(node)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/jinja2/visitor.py", line 38, in visit
    return f(node, *args, **kwargs)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/jinja2/compiler.py", line 754, in visit_Template
    self.blockvisit(node.body, frame)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/jinja2/compiler.py", line 378, in blockvisit
    self.visit(node, frame)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/jinja2/visitor.py", line 38, in visit
    return f(node, *args, **kwargs)
  File "/Users/user_name/Documents/project_name/test/lib/python3.7/site-packages/jinja2/compiler.py", line 1044, in visit_For
    if node.test:
AttributeError: 'For' object has no attribute 'test'

I am using the latest versions of moto and boto3.

What could potentially be the issue here?

you should share how you are setting up the tests as well – gold_cy Mar 15 '19 at 19:36 — gold_cy, Mar 15 '19 at 19:36
Thank you, added this to the post. – jli24 Mar 15 '19 at 19:48 — jli24, Mar 15 '19 at 19:48

score 0 · Answer 1 · answered Mar 15 '19 at 20:52

Fixed my issue but might be useful to get some clarity from other users on why this is the case.

When running my unit tests from a virtual environment, I experience this error but when running locally outside of a virtual environment, the code works.

I also get the proper response from the create_bucket and put_object boto3 commands when running outside the virtual environment, instead of the response I was receiving before.

Running coverage.py on project with moto causes jinja2 error

1 Answers1