MarkLogic: Failed to update hostname - waiting to retry

Question

I'm using cloudformation template for 3 node cluster and testing DR for AZ failure in AWS.

What I did is:

1. created successfully 3 node cluster (Master +2 slaves),
2. terminated ASG with Master ( instances under ELB go to OutOfService status, which is fine) and EBS that Master is using
3. created ASG with Master & EBS in another AZ (result: Master comes back online, ELB is back, website is back BUT... slaves never leave OutOfService status)
4. I deleted ASG with slave2 & its EBS. Recreated ASG & EBS but Slave instance complaints with these messages:

Mar 3 16:31:30 ip-10-58-3-36 MarkLogic: Original host name ip-10-58-3-68.us-west-1.compute.internal does not match our new host name ip-10-58-3-36.us-west-1.compute.internal Mar 3 16:31:30 ip-10-58-3-36 MarkLogic: [/opt/MarkLogic/mlcmd/scripts/initialize-node.xsh line: 52] Mar 3 16:31:31 ip-10-58-3-36 MarkLogic: No cluster hosts online - trying other hosts Mar 3 16:31:31 ip-10-58-3-36 MarkLogic: [/opt/MarkLogic/mlcmd/scripts/initialize-node.xsh line: 56] Mar 3 16:31:31 ip-10-58-3-36 MarkLogic: No known hosts online Mar 3 16:31:31 ip-10-58-3-36 MarkLogic: [/opt/MarkLogic/mlcmd/scripts/initialize-node.xsh line: 59] Mar 3 16:31:31 ip-10-58-3-36 MarkLogic: Failed to update hostname - waiting to retry Mar 3 16:31:31 ip-10-58-3-36 MarkLogic: [/opt/MarkLogic/mlcmd/scripts/initialize-node.xsh line: 153] Mar 3 16:31:31 ip-10-58-3-36 MarkLogic: Sleeping for 14 seconds Mar 3 16:31:31 ip-10-58-3-36 MarkLogic: [/opt/MarkLogic/mlcmd/scripts/initialize-node.xsh line: 154]

I scrolled through /var/log/messages and I see:

Mar 3 15:05:05 ip-10-58-3-36 MarkLogic: MARKLOGIC_INSTANCE: INSTANCE:i- MDM_NODE_NAME: MARKLOGIC_NODE_NAME: NodeB# MARKLOGIC_ZONE:us-west-1a ZONE: us-west-1b

MARKLOGIC_ZONE parameter is pointing to the wrong zone.

My question is: Why this parameter is wrong here? Is that a bug in the software?

I tried updating this parameter through user_data but still no dice.

I'm curious to see how others resolved that issue because it doesn't look like this cluster is self healing.

Another thing is if I kill Master(stop service/stop ec2) then whole cluster is unavailable. Is it possible to set, so Slaves can promote themselves to become Master?

I'm using MarkLogic 8.0-6

Answer 1

If you are hand-modifying resources created in the template, you will have big problems. The parameter is 'wrong', most likely because of :

created ASG with Master & EBS in another AZ

AWS CF Templates 'manage' all resources they create, deleting and creating new ones will cause internal issues, you should instead 'update' templates so that Cloud Formation is aware of what is going on and handles (if possible) all the dependencies.

The Managed Cluster feature itself is not aware of the CF templates in any way (you can run a managed cluster without using CF). It is configured by environment variables at startup, then persists long term state in DynamoDB so that it can survive any or all EC2 instances being terminated. The initial state needs to be consistent with the dynamic state through their lifetime. Once a new instances runs for the first time, the state initial configuration is cached in the root volume to make sure it doesn't change for the lifetime of that EC2 instance. When using the Sample Templates, these are passed in as user data. EC2 instances started from an ASG require a "Launch Configuration" to supply the User Data. This is setup in the template. If you delete an ASG (or launch config) -- and create a new one -- not only will CF get confused, but unless you very carefully configure the Launch Configuration and user data correctly then nodes that start up will be mis-configured -- at best. Once a new instances runs for the first time, the state initial configuration is cached in the root volume to make sure it doesn't change for the lifetime of that EC2 instance.

That is what you are seeing: nodes getting a configuration from the wrong zone, due to creating an ASG in a different zone but getting a configuration from the original zone. To do this kind of surgery requires a deep understanding of the whole process and architecture, even then its not likely to be successful as it was not designed to have the configuration changed in this way.

It is designed to be fault tolerant in the way you are expecting, but it is a bit more complex due to limitations of EBS -- EBS volumes do not cross zones, therefore starting a ML instance in a new zone wont be able to directly use another's EBS volume. Instead you need to replicate forests across zones first, then if there is a zone failure the other nodes will have replicas and the cluster survives. If you need more nodes then modify the "Instances per Zone" via a CF update. This will spawn new instances in the available zone and they will join the cluster. The Managed Cluster feature operates only up to the 'cluster join' level. You need to manage data (forest, volume etc) rebalancing on your own (via the REST API, manually, or other procedures).

In general -- * The Managed Cluster feature is architected to be resilliant against, and by utilizing, EC2 instance termination. If something goes wrong the best way to fix it is to let the instance die and a new one takes its place.

• Make no changes to 'resources' that you didn't not manually create, instead update the CF stack in place. You cannot 'migrate' resources. Instead you create new ones, migrate data, and retire the old.

• Replicate all critical data so that one zone failure does not isolate it, for example the Security DB is required to be accessible at all times by all nodes. With replication, a cluster can survive all zones failing and restart on any zone as it comes up. You just need to let it.

• Work with the ASGs not against them. Read up on various ways of suspending ASG processes so that you can 'rescue' servers if needed without them being killed. If you need more then increase the # of nodes.

• ASG's are configured to have ONE per zone per clusters.