2

We have an application running on 1 or 2 nodes, depending on the environment, with a shared cache based on Hazelcast.

One of the requests on the application triggers about 1000 requests on this cache (all cache hits).

In a single node configuration this works nicely. Specifically each request takes less then 10ms.

But if we use 2 nodes, each cache request takes about 20-200ms. We thought this would be due to Hazelcast getting the data from the remote node, which of course involves network traffic. So we configured it to use a NearCache, which to our understanding should result in about the same access speed as a single local cache. But it doesn't, it doesn't seem to have an effect on the performance at all.

So now I'm wondering:

  • How can I check if the NearCache configuration actually works?
  • How can I get read performance close to a local cache, but updates (asynchronously) communicated to the/all remote caches, which I thought I would get by configuring a NearCache?

We initialize our cache with the following configuration:

HazelcastConfiguration.getOrCreateCache(
                cacheName,
                new CacheConfig<K, CR>()
                        .setNearCacheConfig(new NearCacheConfig())
                        .setExpiryPolicyFactory(
                                HazelcastConfiguration.createExpiryPolicyFactory(expiryAfterModification)),
                cacheManager
        );
Jens Schauder
  • 77,657
  • 34
  • 181
  • 348

1 Answers1

1

It depends on a lot of things, but 20/200ms is a lot. When we run a 4 node cluster benchmark (4 dual socket xeon machine) on 1 standard 1 GbE network, 99 percentile is far below 1 ms. We are talking here without nearcache.

Near cache should give very good performance because if the data is locally cached, then the call is local + the whole internal infrastructure for operation execution is skipped (at least for the IMap and the near cache; not completely sure about the Cache.

Can you try hitting the same set of key(s)? Just to make sure you are hitting the near cache.

What are you using as key and value? Are they large or serialization unfriendly?

I you are still running into problems; quickly try to IMap and enable near cache and check if you see big differences.

pveentjer
  • 10,545
  • 3
  • 23
  • 40
  • Thx, for confirming that this is incredibly slow. We actually have only one key (a String). The value itself is a Map with a couple thousand entries (string key and simple Object value). So it is probably rather large. – Jens Schauder Jun 17 '15 at 12:59
  • Ahhh... there comes the monkey out of the sleave (Dutch proverb). A likely suspect of your performance problem is that this big value needs to be fully deserialized on every read.. Can you try switching to HZ 3.5 and set OBJECT in memory format on the near cache. If I remember correctly there is an optimization that should skip the deserialization on the near cache. – pveentjer Jun 18 '15 at 05:57
  • If that doesn't solve the problem, then perhaps switching to a faster serialization mechanism. Perhaps by subclassing the hashmap and use one of the Hazelcast serialization API's like DataSerializable. Java serialization is quite slow. – pveentjer Jun 18 '15 at 06:00
  • I actually switched to the OBJECT in memory format, before leaving the office yesterday. I'll get the results today. – Jens Schauder Jun 18 '15 at 06:03
  • switching to OBJECT in memory format had no effect, but I hadn't switched to 3.5. I'm trying to switch to 3.5 now, but have problems with configuring the NearCache http://stackoverflow.com/q/30913938/66686 – Jens Schauder Jun 18 '15 at 11:21
  • Keep us posted. If that doesn't solve the problem; make the value a lot smaller since it is currently very big. If that does give a big performance improvement, then you know the cause of the problem. – pveentjer Jun 19 '15 at 06:09
  • Restructured the cache usage, have now more, much smalle Objects (a few thousand objects, each consisting of about 2 short strings and two dates) access time dropped now to 10 ms which is usable, but far from what I'd expected. Near Cache with Object Format seems to have only a small effect. (20% maybe?). I'm back to 3.4 since I don't know how to configure the nearcache without client in 3.5 (see the other question) – Jens Schauder Jun 19 '15 at 18:40
  • The performance is still pretty bad. Like I said; in our test lab for a remote call the 99th percentile is far below 1ms; this is with near cache disabled. So with near cache enabled, it should be a lot better, since there is no remoting. I'll have a look tomorrow at the implementation. – pveentjer Jun 19 '15 at 19:53
  • Are you still relying on serializable? Please try the DataSerializable interface from HZ. – pveentjer Jun 19 '15 at 19:58