0

In my setup I have two instances of an application that are both up and processing requests at the same time, using Hazelcast to share state and locking. The setup is default in that all the data is on both nodes with 50% being live, 50% being a backup for the other node.

I wish to add blue/green deployment to my setup such that, I can deploy a new version with no interruptions in processing requests and without loosing any data/processing requests erroneously.

During a deploy, I would bring down one instance of the application, meaning the remaining instance will now have 100% of the live data and process requests with that. If anything were to go wrong with that instance, I'd loose all my data (I have no backup setup) - I'm happy accepting this risk.

Once I bring up the new version instance of the application, Hazelcast should start to rebalance the cache, such that its 50% live and 50% backup on each instance again. It is at this point I want to block and wait until the rebalance is fully complete before I advertise the new instance as live. When complete, the new instance has 50% of the live data and 50% as backup.

When rebalancing is complete, I can then bring down the second version, bring up the new version, Hazelcast would initialize a rebalance, I'd block, wait for the rebalance to complete, then advertise the second instance as live.

My blue/green deployment would now be complete.

The question: how do I block the new instance until the rebalance in complete (it's received all the data)?

Cheetah
  • 13,785
  • 31
  • 106
  • 190

1 Answers1

0

Hazelcast automatically rebalances and creates backups as needed as you add/remove members, you don't need to do any additional configuration. Important point is to do graceful shutdowns to avoid data loss during rollover of members.

Ways to shut down a member gracefully:

  • You can call the method HazelcastInstance.shutdown() programatically.
  • You can use JMX API’s shutdown method. You can do this by implementing a JMX client application or using a JMX monitoring tool (like JConsole).
  • You can set the property hazelcast.shutdownhook.policy to GRACEFUL and then shutdown by using kill -15 . Your member will be gracefully shutdown.
  • You can use the "Shutdown Member" button in the member view of Hazelcast Management Center if you're using it.

OTOH, Hazelcast wise you don't need to block a member during this process. Is this an application specific requirement?

sertug
  • 872
  • 6
  • 10
  • Hmmm....after I've brought down an instance (so there is one live instance left) and brought up a new one. How do I know the data synchronization is complete before bringing down the remaining live instance to ensure there is no data loss. Are you suggesting that Hazelcast itself will block on graceful shutdown until this process is complete? How does Hazelcast know to expect anything member? – Cheetah Dec 03 '18 at 08:34
  • Yes, during a graceful shutdown Hazelcast ensures all this process is done before safely shutting down. Also, during migration (or data sync as you say) only a subset of write operations are blocked automatically but rest of operations are allowed, so you don't have to block anything yourself. – sertug Dec 03 '18 at 10:55