0

We are using MarkLogic 9.0.8.2

It was working fine and then suddenly when i was configuring failover for security database with master & replica forests on multiple nodes, i started getting issue with not accepting credentials.

I looked at logs and found that security database state is open

2019-07-03 09:48:36.465 Info: Forest security-01 state changed from recovering to open
2019-07-03 12:45:49.688 Info: Mounted forest security-02-replica remotely on ML03
2019-07-03 09:48:36.465 Info: Forest security-01 state changed from recovering to open
2019-07-03 12:45:49.691 Info: Mounted forest security-03 remotely on ML03
2019-07-03 12:45:49.699 Info: Mounted forest security-01-replica remotely on ML02
2019-07-03 12:45:49.700 Info: Mounted forest security-02 remotely on ML02.
2019-07-03 12:45:52.055 Info: Database Security is online with 3 forests

So looking at log, it seems, everything is fine.

so i am not sure why it is not accepting the credentials. How can i retrieve or reset admin password?

I think, i got the cause, i have followed below steps to configure failover on security database

  • Created 3 master forests
  • Created 3 replicas
  • Attached new forests to database, clicked on attach & retire on old by default security forest
  • on configuration machine, deselect old security DB and kept selection on retire checkbox And this is where i think problem is, i did not waited for merging to finish :( MarkLogic service is running, will not merge in background?

I have already restarted service, VMs multiple times.

Now question, how to solve (already loaded around 15M data) this? How to attach old security forest for merge to happen

enter image description here

Manish Joisar
  • 1,256
  • 3
  • 23
  • 47
  • Were you using a tool like ml-gradle to do this configuration? Is it possible you changed your password via such a tool unintentionally? – Rob S. Jul 03 '19 at 14:16
  • Thanks Rob S for your reply, No, was doing thru MarkLogic Admin Interface only. And as i am working on jumpbox, so always use copy paste for credentials. BTW, it was working fine till this morning – Manish Joisar Jul 03 '19 at 14:28
  • Added more info on possible causes in original question – Manish Joisar Jul 04 '19 at 05:45

1 Answers1

1

From MarkLogic's knowledgebase

Multiple forests for Security database

We do not recommend configuring multiple forests for the Security database, as this can cause failover issues when doing upgrades and restarts. Security database should have a single primary forest and replicas on all hosts to ensure High Availability.

https://help.marklogic.com/Knowledgebase/Article/View/601/0/multiple-forests-for-security-database

It seems possible that you may have detached the security database before the documents were migrated to the other forests. MarkLogic maintains a 'databases.xml' file in the configuration folder that contains details about the forests that are attached among other things. Older versions are also present with a number suffix eg. 'databases_1.xml', 'databases_2.xml'

You could try the following on each node:

  1. Stop ML service
  2. Navigate to the ML's config directory and rename 'databases.xml' to 'databases_bad.xml'
  3. Rename 'databases_1.xml' to 'databases.xml'

Finally, restart ML service on all the nodes and wait for the existing documents to be migrated.

PS: I haven't seen this mentioned on ML knowledge base so I am not sure if this is the recommended approach but I have tried this locally and was able to restore to a previous configuration by renaming the database files.

Community
  • 1
  • 1
Arnold
  • 28
  • 7
  • Thanks Arnold for your quick response, so now need to see how can i retrieve admin credentials else i need to start from zero. I can only revert only when admin interface is available. – Manish Joisar Jul 03 '19 at 13:45
  • 1
    @ManishJoisar - I have updated my answer with a potential fix – Arnold Jul 04 '19 at 07:11
  • You may want to reach out to Support, and have them guide you with this. It is easy to mess up when changing those xml config files. – grtjn Jul 05 '19 at 07:32
  • Thanks Arnold, I was able to follow the steps mentioned by you and can see all forest including security forest which i retired and detached immediately before it move all its data to other forest. So after data moved, i retired security forest and now everything is fine. Thanks for your solution. – Manish Joisar Jul 14 '19 at 16:26