6

After searching the internet for a good solution to an embedded Java AWS S3 mock it seemed that S3Ninja and S3Proxy seemed to be the most popular solutions.

However there doesn't seem to be an easy way to fire these up programmatically. After giving up with S3Ninja, I tried to do it with S3Proxy but it's not quite working.

Maven Dependencies

<dependency>
    <groupId>org.gaul</groupId>
    <artifactId>s3proxy</artifactId>
    <version>${s3proxy.version}</version>
    <scope>test</scope>
</dependency>

Code

String endpoint = "http://127.0.0.1:8085";
URI uri = URI.create(endpoint);
Properties properties = new Properties();
properties.setProperty("s3proxy.authorization", "none");
properties.setProperty("s3proxy.endpoint", endpoint);
properties.setProperty("jclouds.provider", "filesystem");
properties.setProperty("jclouds.filesystem.basedir", "/tmp/s3proxy");

ContextBuilder builder = ContextBuilder
        .newBuilder("filesystem")
        .credentials("x", "x")
        .modules(ImmutableList.<Module>of(new SLF4JLoggingModule()))
        .overrides(properties);
BlobStoreContext context = builder.build(BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();

S3Proxy s3Proxy = S3Proxy.builder().awsAuthentication("x", "x").endpoint(uri).keyStore("", "").blobStore(blobStore).build();
s3Proxy.start();

BasicAWSCredentials awsCredentials = new BasicAWSCredentials("x", "x");

AmazonS3Client client = new AmazonS3Client(awsCredentials, new ClientConfiguration());
client.setEndpoint(endpoint);

// Should Throw AWS Client Exception as Bucket / Key does not exist!
GetObjectRequest objectRequest = new GetObjectRequest("bucket", "key");
S3Object object = client.getObject(objectRequest);

s3Proxy.stop();

Exception

java.lang.NoSuchMethodError: com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.<init>(Lcom/google/gson/internal/ConstructorConstructor;Lcom/google/gson/FieldNamingStrategy;Lcom/google/gson/internal/Excluder;)V

at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory.<init>(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:116)
at org.jclouds.json.config.GsonModule.provideGson(GsonModule.java:129)

...

at org.jclouds.providers.config.BindProviderMetadataContextAndCredentials.backend(BindProviderMetadataContextAndCredentials.java:84)

...

at org.jclouds.ContextBuilder.build(ContextBuilder.java:581)

Any help is truly appreciated. I'm sure this is a big requirement for many Java Integration Tests that interact with AWS S3.

Andrew Gaul
  • 2,296
  • 1
  • 12
  • 19
ptimson
  • 5,533
  • 8
  • 35
  • 53
  • 1
    If it is an integration test, don't you want it to be talking to the real S3? – Rob Aug 26 '16 at 15:59
  • Not really no. Our integration tests should be able to be run locally without any other component decencies. We use ActiveMQ to mock SQS and embedded Redis for ElasticCache. – ptimson Aug 26 '16 at 16:01
  • 1
    @ptimson To support Rob's comment, your integration test environment should be identical to the production environment. It sounds more like a developer environment than an integration environment. There are mock SQS services for testing. Check out https://github.com/adamw/elasticmq – jzonthemtn Aug 26 '16 at 16:12
  • @jbird Sorry I meant we use ElasticMQ not ActiveMQ. And https://github.com/kstyrc/embedded-redis for ElasticCache. We are looking for an S3 equivalent. This is not an environment it is a test that will be run with JUnit. The purpose of the question was to identify how to setup an embedded S3 like ElasticMQ & Embedded Redis. – ptimson Aug 26 '16 at 16:17
  • I think some of us are confused by the context of "integration test" (vs. unit test) combined with mock services. Mock services may well be a good idea for unit tests, but I would argue that mocks are misguided for an integration test. Having said that, we have found that AWS services are an acceptable dependency even for unit tests (although such unit tests are rare in practice). Sorry, but because our approach is different than yours, I do not have an answer to your question. But this might provide a clue as to why you are not getting helpful answers. – Rob Aug 26 '16 at 17:30
  • @Rob Perhaps I should have called them Service Tests http://www.schibsted.pl/blog/mocking-amazon-sqs-with-elasticmq/ – ptimson Aug 26 '16 at 18:54

2 Answers2

2

Just to comment the reason is because your project is using a conflicting version of gson. S3Proxy's dep requires gson 2.5.

Slava Markeyev
  • 317
  • 1
  • 4
  • 13
0

Maybe you give ladon-S3-server a chance. Take a look at my github reference.

The core is based on a servlet and has very little dependencies.

Ole Pannier
  • 3,208
  • 9
  • 22
  • 33
RalfU
  • 31
  • 2