Is it possible to start a PIvotal GemFire Server, Locator and Client in one JVM?

Question

I want to start a Pivotal GemFire Server, Locator and client in a single JVM with Spring Boot.

The server and Locator start OK (I used "start-locator" in server gemfire.properties).

However, when I tried to start the client connected to the locator, I got an exception:

java.lang.IllegalStateException: A connection to a distributed system already exists in this VM. 

    at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.validateSameProperties(InternalDistributedSystem.java:3054)
    at com.gemstone.gemfire.distributed.DistributedSystem.connect(DistributedSystem.java:1642)

And, in the Exception, the existing properties that server used to connect to the Locator is printed out.

Here is the code at client:

@Bean(name = "GemfireClientProperties")
Properties gemfireClientProperties() {
    Properties gemfireProperties = new Properties();
    gemfireProperties.setProperty("mcast-port", "0");
    gemfireProperties.setProperty("log-level", "config");
    return gemfireProperties;
}


@Bean(name = "GemfireClientPool")
PoolFactoryBean gemfireClientPool() {
    PoolFactoryBean gemfirePool = new PoolFactoryBean();
    gemfirePool.setRetryAttempts(1);
    gemfirePool.setLocators(Collections.singletonList(new ConnectionEndpoint("localhost", 17202)));
    return gemfirePool;
}


@Bean(name = "clientCache")
ClientCache clientCache(@Qualifier("GemfireClientProperties") Properties gemfireClientProperties) {
    ClientCacheFactory clientCacheFactory = new ClientCacheFactory(gemfireClientProperties);
    return clientCacheFactory.create();
}

Pivotal GemFire version is 8.2.5.

I traced in debug mode, the exception is thrown at the clientCacheFactory.create() line, and in the method, the properties file passed in is correct, which has only 2 entries.

Does Pivotal GemFire have some limitation such that I cannot connect this way?

Answer 1

Yes, this is known limitation, you can't have more than one cache per JVM (other than a server with an embedded locator). Cheers.

Answer 2

In short, NO, you cannot have a peer Cache instance (with embedded Locator) and a ClientCache instance in the same JVM (or Java application process).

In Apache Geode/Pivotal GemFire, the cache instance, whether a peer Cache (a peer member of the distributed system, or cluster), or a ClientCache instance, is a Singleton. Therefore, there can only be 1 GemFire cache instance per JVM (or more technically, ClassLoader, but we will not go down that ugly road).

Anyway, when a cache instance already exists, then any subsequent cache creation attempt say, a ClientCache instance creation using GemFire's o.a.g.cache.client.ClientCacheFactory (when a peer Cache instance, created by the o.a.g.cache.CacheFactory, already exists) will result in a check where GemFire pretty much expects the cache instances to be the same (i.e. a peer Cache exists, therefore better be attempting to create the same (by configuration, anyway) "peer" Cache instance). Clearly a ClientCache instance will not have the same (distributed system) configuration as a peer Cache instance, and there for fails.

You can see the logic beginning here. The first thing that a cache instance creation does is check and lookup an existing cache instance. A ClientCache instance cannot have the locators and mcast-port properties set (here); if they are, it results in an error (usually another diff between client and peer cache instances, particular for the locators property).

If the existing instance is not closed or closing, then it is going to validate the configuration. Pretty much the only way to create another cache instance is when it is the same instance. The configuration validation is quite extensive.

There have been discussion about changing the "Singletone" nature of the GemFire cache instances on the Apache Geode dev lists, but nothing has materialized in this regard yet as it is a major undertaking, partially because the static (Client)CacheFactory.getAnyInstance() (or worse, GemFireCacheImpl.getInstance(); both peer and client cache instances share the same base class... org.apache.geode.cache.internal.GemFireCacheImpl) was used extensively throughout the codebase rather than passing the cache instance to the dependent GemFire objects/components, e.g. like Regions.

Anyway, sorry this answer does not provide you with any relief.

However, I often times create 2 separate Spring Boot application (classes) to configure and bootstrap both a client and 1 or more servers, individually.

On the server, I even use a Spring profile to active a Locator/Manager in a single server instance so that I can form a small cluster (with the embedded Locator) and Manager so that I can also connect up to this cluster using Gfsh.

So, you can use the same class to start a small cluster. The first server would be stated with:

$java -cp ... -Dspring.profiles.active=locator-manager example.app.geode.cache.server. GeodeServerApplication

And subsequence servers, joining the first server, would be started using:

$java -cp ... -Dspring.data.gemfire.name=ServerTwo -Dspring.data.gemfire.cache.server.port=42424 example.app.geode.cache.server. GeodeServerApplication

NOTE, you must be careful to very the member name (using the spring.data.gemfire.name property since GemFire requires peer member names to be unique) and you must vary the CacheServer port (used by cache clients to connect; hence the spring.data.gemfire.cache.server.port property) otherwise you will hit a java.net.BindException, since by default the CacheServer listens on port 40404.

The well-known SDG (System) properties are the easiest way to vary the configuration and get a small cluster running (without Gfsh even; cool!).