0

We are running Ignite cluster with 12 nodes running on Ignite 2.7.0 on openjdk 1.8 at RHEL platform.

Seeing heavy cputime spent with https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#replace-K-V-V-

We are witnessing slowness with one of our process and when we tried to drill it further by profiling the JVM, the main culprit (taking ~78% of total time) seems to be coming from Ignite cache.repalce(K,V,V) api call.
Out of 77.9 by replace, 39% is taken by GridCacheAdapater.equalVal and 38.5% by GridCacheAdapter.put

Cache is Partitioned and ATOMIC with readThrough,writeThrough,writeBehindEnabled set to True.

Attaching the profiling snapshot of one node(similar is the profiling result on other nodes), Can someone please check and suggest what could be the cause OR some known performance issue with this Ignite version related to cache.replace(k,v,v) api ?

enter image description here JVM Prolfiling Snapshot of one node

tarunk
  • 549
  • 2
  • 7
  • 17

1 Answers1

1

I guess that it can be related to next issue:

https://issues.apache.org/jira/browse/IGNITE-5003

The problem there related to the operations for the same key before the previous batch of updates (that contains this key) will be stored in the database.

As I see it should be added to Ignite 2.8.

Update:

I tested putAll operation. From the next two pictures you can see that putAll waiting for GridCacheWriteBehindStore.write (two different threads) that contains updateCache:

public void write(Entry<? extends K, ? extends V> entry) {
    try {
        if (log.isDebugEnabled())
            log.debug(S.toString("Store put",
                "key", entry.getKey(), true,
                "val", entry.getValue(), true));

        updateCache(entry.getKey(), entry, StoreOperation.PUT);
    }

enter image description here

enter image description here

And provided issue can affect your put operations (or replace as well).

Andrei Aleksandrov
  • 686
  • 4
  • 10
  • Thanks @Andrei Do you have some more details on how this ticket would be impacting cache.replace to behave slow. Actually while I am expanding the profiling result further I am seeing time if divided in GridCacheAdapater.equalVal and GridCacheAdapter.put which in turn don't seems to be calling GridCacheWriteBehindStore.updateCache anywhere in profiling result. – tarunk Jan 29 '20 at 13:43
  • 1
    Please take a look at the updated answer. You don't see it because there are two different threads. However, probably you see another issue that isn't related to the current JIRA ticket. – Andrei Aleksandrov Jan 29 '20 at 15:09
  • Thanks Andrei for more insight, this could be the reason. Hoping someone from Ignite Dev community can confirm this to be sure. – tarunk Jan 30 '20 at 10:13