How to restore cache after ignite server reconnected

Question

Really appreciate if someone can help me out.

I have an ignite server written in Java, and have a client written in C#, the client can be connected to the server, and can get server's cache correctly. once the server restarted, the client received the EVT_CLIENT_NODE_RECONNECTED event from server. But the cache cannot be used any more.

Server code:

CacheConfiguration cacheConfiguration = new CacheConfiguration();

    cacheConfiguration.setName("Sample");
    cacheConfiguration.setCacheMode(CacheMode.REPLICATED);
    cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
    cacheConfiguration.setRebalanceMode(CacheRebalanceMode.ASYNC);
    cacheConfiguration.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
    cacheConfiguration.setBackups(0);
    cacheConfiguration.setCopyOnRead(true);
    cacheConfiguration.setStoreKeepBinary(false);

    cacheConfiguration.setReadThrough(false);
    cacheConfiguration.setWriteThrough(true);
    cacheConfiguration.setWriteBehindEnabled(true);
    cacheConfiguration.setWriteBehindFlushFrequency(2000);
    cacheConfiguration.setWriteBehindFlushThreadCount(2);

    DriverManagerDataSource theDataSource = new DriverManagerDataSource();
    theDataSource.setDriverClassName("org.postgresql.Driver");
    theDataSource.setUrl("jdbc:postgresql://192.168.224.128:5432/sample");
    theDataSource.setUsername("postgres");
    theDataSource.setPassword("password");


    CacheJdbcPojoStoreFactory jdbcPojoStoreFactory = new CacheJdbcPojoStoreFactory<Long, SampleModel>()
            .setParallelLoadCacheMinimumThreshold(0)
            .setMaximumPoolSize(1)
            .setDataSource(theDataSource);

    cacheConfiguration.setCacheStoreFactory(jdbcPojoStoreFactory);


    Collection<JdbcType> jdbcTypes = new ArrayList<JdbcType>();

    JdbcType jdbcType = new JdbcType();
    jdbcType.setCacheName("Sample");
    jdbcType.setDatabaseSchema("public");
    jdbcType.setKeyType("java.lang.Long");

    Collection<JdbcTypeField> keys = new ArrayList<JdbcTypeField>();
    keys.add(new JdbcTypeField(Types.BIGINT, "id", long.class, "id"));
    jdbcType.setKeyFields(keys.toArray(new JdbcTypeField[keys.size()]));
    Collection<JdbcTypeField> vals = new ArrayList<JdbcTypeField>();

    jdbcType.setDatabaseTable("sample");
    jdbcType.setValueType("com.nmf.SampleModel");

    vals.add(new JdbcTypeField(Types.BIGINT, "id", long.class, "id"));
    vals.add(new JdbcTypeField(Types.VARCHAR, "name", String.class, "name"));

    jdbcType.setValueFields(vals.toArray(new JdbcTypeField[vals.size()]));

    jdbcTypes.add(jdbcType);

    ((CacheJdbcPojoStoreFactory)cacheConfiguration.getCacheStoreFactory()).setTypes(jdbcTypes.toArray(new JdbcType[jdbcTypes.size()]));


    IgniteConfiguration icfg = new IgniteConfiguration();
    icfg.setCacheConfiguration(cacheConfiguration);

    Ignite ignite = Ignition.start(icfg);

SampleModel:

public class SampleModel implements Serializable {
private long id;
private String Name;

public long getId() {
    return id;
}

public void setId(long id) {
    id = id;
}

public String getName() {
    return Name;
}

public void setName(String name) {
    Name = name;
}

@Override
public boolean equals(Object o) {
    if (this == o) return true;
    if (!(o instanceof SampleModel)) return false;

    SampleModel that = (SampleModel) o;

    return id == that.id;
}

@Override
public int hashCode() {
    return (int) (id ^ (id >>> 32));
}

}

Client Code:

ExecutorService executor = Executors.newSingleThreadExecutor(r -> new Thread(r, "worker"));


    CacheConfiguration cacheConfiguration = new CacheConfiguration();

    cacheConfiguration.setName("Sample");
    cacheConfiguration.setCacheMode(CacheMode.REPLICATED);
    cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
    cacheConfiguration.setRebalanceMode(CacheRebalanceMode.ASYNC);
    cacheConfiguration.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
    cacheConfiguration.setBackups(0);
    cacheConfiguration.setCopyOnRead(true);
    cacheConfiguration.setStoreKeepBinary(false);

    IgniteConfiguration icfg = new IgniteConfiguration();
    icfg.setCacheConfiguration(cacheConfiguration);

    icfg.setClientMode(true);

    final Ignite ignite = Ignition.start(icfg);

    ignite.events().localListen(new IgnitePredicate<Event>() {
        public boolean apply(Event event) {
            if (event.type() == EVT_CLIENT_NODE_RECONNECTED) {
                System.out.println("Reconnected");

                executor.submit(()-> {
                    IgniteCache<Long, SampleModel> cache = ignite.getOrCreateCache("Sample");

                    System.out.println("Got the cache");

                    SampleModel model = cache.get(1L);

                    System.out.println(model.getName());
                });
            }

            return true;
        }
    }, EVT_CLIENT_NODE_RECONNECTED);

    IgniteCache<Long, SampleModel> cache = ignite.getOrCreateCache("Sample");

    SampleModel model = cache.get(1L);

    System.out.println(model.getName());

Error log on Client:

    SEVERE: Failed to reinitialize local partitions (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], nodeId=dea5f59b, evt=DISCOVERY_CUSTOM_EVT]
class org.apache.ignite.IgniteCheckedException: Failed to start component: class org.apache.ignite.IgniteException: Failed to initialize cache store (data source is not provided).
    at org.apache.ignite.internal.util.IgniteUtils.startLifecycleAware(IgniteUtils.java:8726)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCache(GridCacheProcessor.java:1486)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1931)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1833)
    at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:379)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeFuture.java:688)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:529)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1806)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
    at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteException: Failed to initialize cache store (data source is not provided).
    at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.start(CacheAbstractJdbcStore.java:298)
    at org.apache.ignite.internal.util.IgniteUtils.startLifecycleAware(IgniteUtils.java:8722)
    ... 9 more

July 25, 2017 12:58:38 PM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to wait for completion of partition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=dea5f59b-bdda-47a1-b31d-1ecb08fc746f, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.224.1, 192.168.6.15, 192.168.80.1, 2001:0:9d38:90d7:c83:fac:98d7:5fc1], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, Ares-W11/169.254.194.93:0, /2001:0:9d38:90d7:c83:fac:98d7:5fc1:0, /192.168.6.15:0, windows10.microdone.cn/192.168.224.1:0, /192.168.80.1:0], discPort=0, order=2, intOrder=0, lastExchangeTime=1500958697559, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6, isClient=true], topVer=2, nodeId8=dea5f59b, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1500958718133]], crd=TcpDiscoveryNode [id=247d2926-010d-429b-aef2-97a18fbb3b5d, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.224.1, 192.168.6.15, 192.168.80.1, 2001:0:9d38:90d7:c83:fac:98d7:5fc1], sockAddrs=[/192.168.6.15:47500, /2001:0:9d38:90d7:c83:fac:98d7:5fc1:47500, windows10.microdone.cn/192.168.224.1:47500, /192.168.80.1:47500, Ares-W11/169.254.194.93:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1500958718083, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], nodeId=dea5f59b, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false, hash=842035444], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=true, clientOnlyExchange=false, initTs=1500958718133, centralizedAff=false, changeGlobalStateE=null, exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0, remaining=[247d2926-010d-429b-aef2-97a18fbb3b5d], srvNodes=[TcpDiscoveryNode [id=247d2926-010d-429b-aef2-97a18fbb3b5d, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.224.1, 192.168.6.15, 192.168.80.1, 2001:0:9d38:90d7:c83:fac:98d7:5fc1], sockAddrs=[/192.168.6.15:47500, /2001:0:9d38:90d7:c83:fac:98d7:5fc1:47500, windows10.microdone.cn/192.168.224.1:47500, /192.168.80.1:47500, Ares-W11/169.254.194.93:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1500958718083, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteCheckedException: Failed to start component: class o.a.i.IgniteException: Failed to initialize cache store (data source is not provided)., hash=1281081640]]
class org.apache.ignite.IgniteCheckedException: Failed to start component: class org.apache.ignite.IgniteException: Failed to initialize cache store (data source is not provided).
    at org.apache.ignite.internal.util.IgniteUtils.startLifecycleAware(IgniteUtils.java:8726)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCache(GridCacheProcessor.java:1486)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1931)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1833)
    at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:379)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeFuture.java:688)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:529)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1806)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
    at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteException: Failed to initialize cache store (data source is not provided).
    at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.start(CacheAbstractJdbcStore.java:298)
    at org.apache.ignite.internal.util.IgniteUtils.startLifecycleAware(IgniteUtils.java:8722)
    ... 9 more

score 0 · Answer 1 · answered Jul 20 '17 at 07:59

0

Only server nodes store caches(except of Local caches), so, when you restarted server node, this cache was stopped. The problem here is that client node was reconnected to the cluster, but not joined it as new node. That's why cache was not created again.

I think it's wrong behavior and cache should be recreated at client reconnecting. I've created an issue for that.

As a workaround, you can use Ignite.GetOrCreateCache("Sample") method instead of Ignite.GetCache("Sample")

answered Jul 20 '17 at 07:59

Evgenii Zhuravlev

2,987
1
9
15

thanks for your reply, seems it will be hang when calling the code `Ignite.GetOrCreateCache("Sample") `, update the question by adding more error log. – Ares Zhu Jul 20 '17 at 09:26
Please share also CacheConfiguration – Evgenii Zhuravlev Jul 20 '17 at 10:42
Yes, also update the CacheConfiguration in the question description. – Ares Zhu Jul 20 '17 at 11:12
Why did you configure KeepBinaryInStore = true? You don't have configured cacheStore – Evgenii Zhuravlev Jul 20 '17 at 13:37
Sorry, Im not expert on Ignite, may I know how to set configured cacheStore? – Ares Zhu Jul 20 '17 at 14:52
First of all, are you sure that you need persistence? If not, just remove KeepBinaryInStore = true. If you need it, check this doc: https://apacheignite-net.readme.io/v2.0/docs/persistent-store – Evgenii Zhuravlev Jul 20 '17 at 14:56
or, you can find an example of xml config for cache store here: https://stackoverflow.com/questions/44819513/org-apache-ignite-ignitecheckedexception-cannot-enable-write-through/44819881#44819881 – Evgenii Zhuravlev Jul 20 '17 at 14:56
the cache mode is Replicated, so the it is stored on the server not the client, the client is only to get the cache instead of create. – Ares Zhu Jul 21 '17 at 04:34
The cache store on the client is required for TRANSACTIONAL caches. For ATOMIC caches (this is the default mode), it's not needed. So, if you use Atomic cache, just delete KeepBinaryInStore = true – Evgenii Zhuravlev Jul 21 '17 at 06:20
OK, even remove KeepBinaryInStore = true, the issue is still there, BTW everything works if server is not started. – Ares Zhu Jul 21 '17 at 06:42
OK, then provide config from server too – Evgenii Zhuravlev Jul 21 '17 at 06:51
Thanks for your help and patience, already updated the cache configuration on the server side. – Ares Zhu Jul 21 '17 at 07:49
Hi Evgenii, do you have any further suggestion or finding? – Ares Zhu Jul 24 '17 at 02:23
have you tried to delete cacheConfiguration.setStoreKeepBinary(true); from server configuration too? – Evgenii Zhuravlev Jul 24 '17 at 05:35
The server uses cacheConfiguration.setStoreKeepBinary(true), since it has persistent store. – Ares Zhu Jul 24 '17 at 06:14
It seems that we cannot remove the `cacheConfiguration.setStoreKeepBinary(true)` on server side. – Ares Zhu Jul 24 '17 at 07:34
Hi Sir, im really stuck by this issue, could you please help me out, I can provide any further information if needed. – Ares Zhu Jul 25 '17 at 02:00
I have paste the whole code for both client and server on the question description, and remove `cacheConfiguration.setStoreKeepBinary(true)` on both sides. – Ares Zhu Jul 25 '17 at 05:12

score 0 · Answer 2 · answered Jul 22 '17 at 18:00

0

Are you still having issues where Ignite.GetOrCreateCache("Sample") hangs? Make sure you aren't making that call from a thread in the System Pool. I was listening for the EVT_CLIENT_NODE_RECONNECTED event and calling Ignite.GetOrCreateCache("Sample") when I ran into a similar issue. For more information, see the answer to this question: Closures stuck in 2.0 when try to add an element into the queue

answered Jul 22 '17 at 18:00

satsie

399
3
9

thanks for your answer, but no lucky, I start a new tread when calling `Ignite.GetOrCreateCache("Sample")` – Ares Zhu Jul 24 '17 at 02:16
In your client code it doesn't look like you are using a separate thread pool. Try something like this: `ignite.compute().withExecutor("my-thread-pool").runAsync(() -> { Ignite.GetOrCreateCache("Sample"); });` Don't forget to set up your custom "my-thread-pool" in the Ignite configuration. – satsie Aug 01 '17 at 03:11

How to restore cache after ignite server reconnected

2 Answers2

Linked