Amazon Kinesis + Integration Tests

Question

I'm currently working on a series of web-services which we need to integrate with Kinesis - the implementation has been done, however we have a series of integration tests (our web-services are all using Spring Boot so we use the @WebIntegrationTest annotation on our test classes to start up a local instance of the server and then call our resources with a TestRestTemplate) which are currently trying and failing to connect to the real Kinesis.

Although in ordinary unit tests it's not a problem to mock out calls to the methods within the Kinesis library, we can't really do this in the integration tests as the whole application stack is wired up with Spring. For a few other things (such as OAuth2 and calls to our other web-services) we've been able to use WireMock to mock out the actual endpoints - what I'd really like to do is use WireMock in this fashion to mock out the call to the AmazonKinesisClient but I can't find any advice on how to do this.

Alternatively I have seen that some AWS components have test libraries written by third parties which allow you to run a local version of it (e.g.: DynamoDbLocal) but can't find such a solution for Kinesis.

Is anyone able to give me some advice on how to run integration tests with Kinesis?

I'm unfamiliar with WebIntegrationTests, but, could you: Create a non-production test stream with a single shard before your tests, run the test, and close it after? Or create a thin wrapper around the aws api on your put/get calls which could funnel objects through a queue? — Chris Riddell, Jun 23 '15 at 02:27
I also ran into this "problem". One more use case that I need is that every developer should be able to run integration tests from local environment. I don't want to create streams for every dev. — Mantas, Jul 13 '15 at 13:37

score 11 · Answer 1 · answered Jan 29 '18 at 11:48

It might already be too late to give the solution but I will add what my team has done to replicate AWS resources locally as we use a lot of Kinesis, DynamoDb, S3 and cloudWatch.

We have created wrappers around Localstack -> https://github.com/localstack/localstack that allow us to spin up local instances of the necessary services as docker containers using docker-compose.

A typical docker-compose.yml file for us looks like:

version: '2'
services:
  localstack:
    image: "localstack/localstack"
    environment:
      - SERVICES=kinesis,dynamodb,cloudwatch
    ports:
      - "4568"
      - "4569"
      - "4582"

Then during the setup phase for the integration-tests, our wrapper fires up docker-compose up and runs the tests against the local infrastructure. Later during tear-down, the wrapper kills the containers.

score 1 · Accepted Answer · answered Sep 30 '15 at 12:07

I ran into the same issue and the only mock implementation I found so far was a nodejs one: https://github.com/mhart/kinesalite It does the trick - I managed to run my Java Kinesis client against it, just had to set the endpoint on the kinesis.properties:

kinesisEndpoint=http://localhost:4567

The downside is that it is not trivial to use it during build time tests - need to figure a way to start the mock kinesis before the test (using a maven plugin or something), didn't get to it yet..

madhead · Answer 3 · 2022-01-17T09:28:11.913

Just a small addition to the existing answers. BTW, they are great, you should really use tools like localstack to start fake AWS services before the test during the test phase.

If you're using JUnit 5 in your tests, your life could be even simpler with JUnit 5 extensions for AWS, a few JUnit 5 extensions that could be useful for testing AWS-related code. These extensions can be used to inject clients for AWS service mocks provided by tools like localstack. Both AWS Java SDK v 2.x and v 1.x are supported:

@ExtendWith(DynamoDB.class)
class AmazonDynamoDBInjectionTest {
    @AWSClient(
        endpoint = Endpoint.class
    )
    private AmazonDynamoDB client;

    @Test
    void test() throws Exception {
        Assertions.assertNotNull(client);

        Assertions.assertEquals(
            Collections.singletonList("table"),
            client.listTables().getTableNames().stream().sorted().collect(Collectors.toList())
        );
    }
}

Here, client will be just injected in your test class and configured according to the Endpoint configuration class.

Amazon Kinesis + Integration Tests

3 Answers3