I'm working on a system that is using Embedded hazelcast in Spring apps running on Kubernetes (OpenShift) and I'm wondering how to properly configure things so things work with scaling up/down, restart of pods etc. I realize I should be using the Hazelcast Kubernetes plugin - but I'm unsure about the following aspects:
Should I do something special about readiness to report Hazelcast readiness? Things that could be done: Explicitly test if Hazelcast is working (by doing something in e.g. a map) or by directly checking the status of the Hazelcast readiness probe? Or is there a third option (maybe not do anything at all - note the application will only report UP once the whole app has initialized including the initial Hazelcast startup).
Do I need to install something in the way of shutdonw hooks to properly shutdown hazelcast? Or does Hazelcast itself react to SIGTERM etc. Documentation I've read seems to suggest the latter is not the case of out-of-the-box, suggesting either Hazelcast configuration needs to be changed to do this, or an explicit shutdown hook installed.
I'm a bit wary regarding #1, especially if it's wise to include an explicit check if Hazelcast is working (accessing a map etc.) because how will this play along with Hazelcast migration and discovery? Will the other peers only discovre the node when the readiness in this sense is UP, and may even kick the node out if it reports DOWN? Then this could give rise to bad feedback efects with nodes entering/exiting the cluster. I'm thinking maybe it's safer to relay the informtion in the Hazelcast readiness probe? Or maybe there's a different way in this specific setting of Kubernetes/Hazelcast?
Tried with out of box configuration but I'm seeing instability when scaling pods up and down.
