I am using Couchbase cache. There is one couchbase server. I have multiple servers to host the app and follow a slight modification of the blue green deployment strategy to deploy app on all the servers.
Deployment strategy - Consider first 50% servers as A and rest as B.
1. Detach A from ELB, B is still serving the requests.
2. Deploy app on A.
3. Reconnect A to ELB.
4. Detach B from ELB but B has some requests to process.
5. Invalidate cache.
6. New requests go to A while B completes requests that it had pending.
7. Post completing pending requests, start deployment on B.
8. Post deployment on B, reconnect B to ELB.
The problem rises with the cache in the Couchbase server. While B has pending requests that run on old code, A is all set to process new requests running on new code. The cache server contains no cached data after step 5.
But in step 6, B will load data into cache according to old code. If A uses that cache, the app will break on A.
If in step 6, A loads data into cache and B uses that cached data, the app will again break on B.
How can I solve this crucial section problem where both A and B are serving requests simultaneously? Any change in deployment strategy is also welcome.