I'm running an Apache Ignite 2.14.0 cluster with persistence that consists of 3 nodes. The nodes are running in a Kubernetes cluster as a stateful set of the following pods: ignite-ss-0, ignite-ss-1 and ignite-ss-2.
In one of my caches I needed to change the type of the key field from int to string. I made the changes in the code, deployed it and then reset the cache with the following command
control.sh --cache destroy --caches my_cache
After having done that I run my application and it seemed to work. However, soon the first instance started failing with the following error:
[17:30:26,445][SEVERE][rebalance-#95][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.binary.BinaryObjectException: Cannot find metadata for object with compact footer (Ignite work directory might have been cleared after restart. Make sure that IGNITE_HOME does not point to a temp folder or any other folder that is destroyed/cleared on restarts) [typeId=7327288, IGNITE_HOME='/app']]]
The other instances are working just fine. The app itself also working fine (Ignite configuration assumes 1 backup for each cache).
The question is: how to fix this crash?
Thank you in advance!
UPDATE: I've resolved this by clearining all data and reinitialising all the caches. But still it would be useful to know how to manage schema updates in caches when persistence is on and how to approach such cases when they occur.
