0

We have a 3 node MarkLogic 9 setup in AWS (we have setup ELB auto-scaling group). So, whenever an instance fails the ELB healthcheck, the EBS volume attached to that instance is still attached to the instance (that failed). Because of this, MarkLogic is unable to start in the newly spawned instance. Has anyone came across this and do you have any idea how to resolve this?

P K
  • 162
  • 12
  • Amazon EBS volumes _cannot_ be attached to a terminated Amazon EC2 instance. – John Rotenstein Aug 12 '19 at 21:13
  • I have updated my question John. Opened an AWS support ticket too. Will keep you posted about what they say. – P K Aug 13 '19 at 11:37
  • What do you mean by "MarkLogic is unable to start in the newly spawned instance"? Why is this related to an Amazon EBS volume? Does MarkLogic do something special with EBS volumes? – John Rotenstein Aug 13 '19 at 11:47
  • Yes @JohnRotenstein. MarkLogic will not store it's data on the root volume (/dev/xvda). It stores it's data only on /dev/sdf. More info is available here https://developer.marklogic.com/products/cloud/aws (Section - Launching an AMI with EC2 Console) – P K Aug 13 '19 at 11:54

1 Answers1

1

If an instance fails an Elastic Load Balancing health check, then the load balancer will not send traffic to that instance. It will keep performing the health check and will resume sending traffic if the health check turns successful.

The Load Balancer will not terminate an instance.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Last week an instance failed the healthcheck and we didn't have any notification (cloudwatch or runscope) rules defined for this. So, our cluster was running with only 2 nodes for almost a week. Last wednesday, bulk job was submitted to the MarkLogic cluster and we noticed that job is getting processed slower than normal. Only at that point we realized that the MarkLogic failed to start in newly spawned instance because it didn't have /dev/sdf attached (as it is still attached to the older instance that failed the healthcheck). – P K Aug 13 '19 at 12:01