0

I have a test suite of ~1000 tests. Previously we were using the mock_s3 decorator which only worked because our "integration tests" just imported modules from other services and mocked their functionality. This was always been undesirable but has not been a priority to fix until recently.

I'm attempting to use moto in standalone server mode, hackily starting the service in conftest.py using 

def start_local_moto_server():
    global moto_subprocess
    moto_subprocess = subprocess.Popen(
        ['moto_server', 's3'],
        stdout=subprocess.PIPE, stderr=subprocess.PIPE,
        bufsize=0, universal_newlines=True,
    )
    print 'Waiting for moto local service to start...'
    while True:
        moto_log = moto_subprocess.stderr.readline()
        print moto_log
        if 'Running' in moto_log:
            break

    boto3.resource(
        's3', region_name='us-east-1', endpoint_url='http://127.0.0.1:5000'
    ).create_bucket(Bucket=TEST_S3_BUCKET_NAME)

As the tests progress, any calls to the s3 client become slower and slower, until finally I get

ReadTimeoutError: Read timeout on endpoint URL: "http://localbox:5000/my-bucket/my/key/file.tar"

Running those tests individually causes all of them to pass.

I don't really know where to begin debugging this or understanding the issue, or how to provide more information. Could someone please help identify the problem?

gmoss
  • 1,019
  • 5
  • 17
  • is this a fixture? you reference conftest but nothing about pytest nor a pytest flag – gold_cy Jul 09 '19 at 23:00
  • sorry, it's just called in `pytest_sessionstart`. – gmoss Jul 09 '19 at 23:56
  • so is this a fixture or no? – gold_cy Jul 09 '19 at 23:59
  • no, it's a flask service running in a separate process behind `http://localhost:5000` that I start at the start of the pytest session. The test configuration instructs boto3 to use this as the `endpoint_url` for the s3 client. – gmoss Jul 10 '19 at 00:15

1 Answers1

1

Answer to slowness provided by the moto developer here: https://github.com/spulec/moto/issues/2288

gmoss
  • 1,019
  • 5
  • 17