2

Problem scenario: The Ignite server of a node. Start a standard client, create a Table, and continuously insert data into the Table using the Insert statement. At the same time as inserting data, another standard client started and deleted the table.

Problem phenomenon: The program executing the Insert statement operation is stuck and there are no error returns.

Error content: This error may not necessarily occur, but there are two possible scenarios: 1.The first scenario: Most of the time, programs executing Insert statements will report the following error:

Exception in thread "main" javax.cache.CacheException: Failed to execute DML statement [stmt=INSERT INTO PRODUCT (ID, NAME, PRODUCTCODE, DISTRICT, NUMBER) VALUES (?, ?, ?, ?, ?);, params=[52043, product52043, p52043, a product, 1000000]]
    at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:829)
    at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:762)
    at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:385)
    at com.example.batchkazhu2.example.Example01.main(Example01.java:32)
Caused by: class org.apache.ignite.internal.processors.query.IgniteSQLException: Failed to execute DML statement [stmt=INSERT INTO PRODUCT (ID, NAME, PRODUCTCODE, DISTRICT, NUMBER) VALUES (?, ?, ?, ?, ?);, params=[52043, product52043, p52043, a product, 1000000]]
    at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeDml(IgniteH2Indexing.java:1356)
    at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.querySqlFields(IgniteH2Indexing.java:1231)
    at org.apache.ignite.internal.processors.query.GridQueryProcessor$2.applyx(GridQueryProcessor.java:2877)
    at org.apache.ignite.internal.processors.query.GridQueryProcessor$2.applyx(GridQueryProcessor.java:2873)
    at org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:35)
    at org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:3425)
    at org.apache.ignite.internal.processors.query.GridQueryProcessor.lambda$querySqlFields$3(GridQueryProcessor.java:2893)
    at org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuerySafe(GridQueryProcessor.java:2931)
    at org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2867)
    at org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2794)
    at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:814)
    ... 3 more
Caused by: class org.apache.ignite.internal.processors.cache.CachePartialUpdateCheckedException: Failed to update keys (retry update if possible).: [52043]
    at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.onPrimaryError(GridNearAtomicAbstractUpdateFuture.java:420)
    at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.onPrimaryResponse(GridNearAtomicSingleUpdateFuture.java:257)
    at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateResponse(GridDhtAtomicCache.java:3168)
    at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$500(GridDhtAtomicCache.java:151)
    at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:305)
    at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:300)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1150)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
    at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1725)
    at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1332)
    at org.apache.ignite.internal.managers.communication.GridIoManager.access$4800(GridIoManager.java:157)
    at org.apache.ignite.internal.managers.communication.GridIoManager$8.execute(GridIoManager.java:1217)
    at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:54)
    at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:564)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node.
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.UpdateErrors.addFailedKeys(UpdateErrors.java:124)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:339)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1783)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1679)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3146)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:151)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:286)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:281)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1171)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:612)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:413)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:319)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:110)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:309)
        ... 8 more
        Suppressed: class org.apache.ignite.internal.processors.cache.CacheStoppedException: Failed to perform cache operation (cache is stopped): PRODUCT
            ... 20 more

The server does not report any errors.

2.Second scenario: Occasionally, the following errors can be reproduced: The program that executes the Insert statement does not report an error, The server node reported the following error:

四月 03, 2023 5:27:43 下午 org.apache.ignite.logger.java.JavaLogger error
严重: Received message without registered handler (will ignore) [futId=2, writeVer=null, msg=GridNearAtomicSingleUpdateFilterRequest [filter=[o.a.i.i.processors.cache.CacheEntrySerializablePredicate@6ff66670], parent=GridNearAtomicSingleUpdateRequest [key=null, parent=GridNearAtomicAbstractSingleUpdateRequest [nodeId=null, futId=2, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=2], parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=needRes|keepBinary]]]], node=01ece6ae-c041-4325-ae6a-810233d0a684, locTopVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], msgTopVer=AffinityTopologyVersion [topVer=2, minorTopVer=2], desc=null]
Registered listeners:
95945896=[GridNearAtomicAbstractUpdateRequest handler [msgIdx=0], GridNearAtomicUpdateResponse handler [msgIdx=1], GridDhtAtomicUpdateRequest handler [msgIdx=2], GridDhtAtomicUpdateResponse handler [msgIdx=3], GridDhtAtomicDeferredUpdateResponse handler [msgIdx=4], GridDhtAtomicNearResponse handler [msgIdx=5], GridNearAtomicCheckUpdateRequest handler [msgIdx=6]]
四月 03, 2023 5:27:43 下午 org.apache.ignite.logger.java.JavaLogger error
严重: Failed to process failed message: java.lang.NullPointerException
java.lang.NullPointerException
    at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.UpdateErrors.prepareMarshal(UpdateErrors.java:134)
    at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.prepareMarshal(GridNearAtomicUpdateResponse.java:370)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onSend(GridCacheIoManager.java:1239)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1274)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1325)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendResponseOnFailedMessage(GridCacheIoManager.java:652)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processFailedMessage(GridCacheIoManager.java:1065)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:403)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:319)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:110)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:309)
    at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1725)
    at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1332)
    at org.apache.ignite.internal.managers.communication.GridIoManager.access$4800(GridIoManager.java:157)
    at org.apache.ignite.internal.managers.communication.GridIoManager$8.execute(GridIoManager.java:1217)
    at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:54)
    at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:564)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
    at java.lang.Thread.run(Thread.java:748)

test code:


public class StartServer {
    public static void main(String[] args) {
        Ignite ignite = Ignition.start();
    }
}
public class StartClient01 {
    public static void main(String[] args) {
        Ignition.setClientMode(true);
        Ignite ignite = Ignition.start();

        IgniteCache<Object, Object> cache = ignite.getOrCreateCache("dummy");

        cache.query(new SqlFieldsQuery("CREATE TABLE PRODUCT (" +
                "  ID INT(11)," +
                "  NAME CHAR(35)," +
                "  PRODUCTCODE CHAR(10)," +
                "  DISTRICT CHAR(100)," +
                "  NUMBER INT(11)," +
                "  PRIMARY KEY (ID)" +
                ") WITH \"TEMPLATE=REPLICATED, CACHE_NAME=PRODUCT\";"));

        for (int i = 1; i <= 1000000; ++i) {
            cache.query(new SqlFieldsQuery("INSERT INTO PRODUCT (ID, NAME, PRODUCTCODE, DISTRICT, NUMBER) VALUES (?, ?, ?, ?, ?);")
                        .setArgs(i, "product" + i, "p" + i, "a product", 1000000)).getAll();

            System.out.println(i);
        }
    }
}
public class StartClient02 {
    public static void main(String[] args) {
        Ignition.setClientMode(true);
        Ignite ignite = Ignition.start();
        IgniteCache<Object, Object> cache = ignite.getOrCreateCache("dummy");
        cache.query(new SqlFieldsQuery("DROP TABLE  IF EXISTS PRODUCT;")).getAll();
        System.out.println("success..");
        ignite.close();
    }
}

Version: The testing environment is using Gridgain Community Edition 8.8.9. After reviewing the Ignite source code, it should also have the same issue.

From this error message, it can be inferred that it was caused by another client's “Drop” cache operation. However, I believe that when encountering this situation during the Insert operation, it should be returned in a timely manner. Is there any solution?

1 Answers1

0

I don't think there is a solution.

The first exception CacheStoppedException: Failed to perform cache operation (cache is stopped): PRODUCT looks reasonable. At least it's clear what's happening there. Trying to insert data to a dropped table should lead to an exception.

The second one is a room for improvement because of a race condition. Nevertheless, Ignite's internals should not throw NPE, instead it should be rewritten to a human-readable error or something. A JIRA needs to be filed to address it.

IGNITE-19203 has been created.

Alexandr Shapkin
  • 2,350
  • 1
  • 6
  • 10