0

MarkLogic 9.0.8.2 Configured Cluster

It was working fine till we started configuring security forest on cluster.

Now we are not able to access MarkLogic Admin Interface.

Log file shows as follows

ErrorLog.txt

2019-03-28 08:31:28.713 Warning: Forest Security fast query timestamp (15536998638611159) lags commit timestamp (15537609012057850) by 61037344 ms

8001_AccessLog.txt

IP - User [28/Mar/2019:16:05:35 +0000] "GET / HTTP/1.1" 500 1978 - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"

8000_ErrorLog.txt

2019-03-28 15:10:52.020 Info: <error:error xsi:schemaLocation="http://marklogic.com/xdmp/error error.xsd" xmlns:error="http://marklogic.com/xdmp/error" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
2019-03-28 15:10:52.020 Info:+  <error:code>XDMP-SECDB</error:code>
2019-03-28 15:10:52.020 Info:+  <error:name/>
2019-03-28 15:10:52.020 Info:+  <error:xquery-version>1.0-ml</error:xquery-version>
2019-03-28 15:10:52.020 Info:+  <error:message>Security database unavailable</error:message>
2019-03-28 15:10:52.020 Info:+  <error:format-string>XDMP-SECDB: Security database unavailable: XDMP-FORESTMNT: Forest forest-security3 not mounted: disconnected</error:format-string>
2019-03-28 15:10:52.020 Info:+  <error:retryable>false</error:retryable>
2019-03-28 15:10:52.020 Info:+  <error:expr/>
2019-03-28 15:10:52.020 Info:+  <error:data>
2019-03-28 15:10:52.020 Info:+    <error:datum>XDMP-FORESTMNT</error:datum>
2019-03-28 15:10:52.020 Info:+    <error:datum>forest-security3</error:datum>
2019-03-28 15:10:52.020 Info:+    <error:datum>disconnected</error:datum>
2019-03-28 15:10:52.020 Info:+  </error:data>
2019-03-28 15:10:52.020 Info:+  <error:stack>
2019-03-28 15:10:52.020 Info:+    <error:frame>
2019-03-28 15:10:52.020 Info:+      <error:uri>/qconsole</error:uri>
2019-03-28 15:10:52.020 Info:+      <error:xquery-version>1.0-ml</error:xquery-version>
2019-03-28 15:10:52.020 Info:+    </error:frame>
2019-03-28 15:10:52.020 Info:+  </error:stack>
2019-03-28 15:10:52.020 Info:+</error:error>

Disk space

/dev/sda2        30G   19G   11G  65% /
devtmpfs        2.0G     0  2.0G   0% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           2.0G  9.1M  2.0G   1% /run
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sda1       497M  105M  392M  22% /boot
/dev/sdb1       7.8G   36M  7.3G   1% /mnt/resource
tmpfs           394M     0  394M   0% /run/user/1000
tmpfs           394M     0  394M   0% /run/user/994

Memory 

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  4526 root      20   0  396544  24816   5608 S   1.3  0.6   0:33.77 python
  3897 root      16  -4  222944  13608   1440 S   0.3  0.3   0:03.93 auoms
  4760 omsagent  20   0 1396640  55524   6108 S   0.3  1.4   0:08.87 omsagent
  4963 daemon    20   0 3449488 197692  35600 S   0.3  4.9   0:16.46 MarkLogic
 11441 idmladm+  20   0  162012   2296   1596 R   0.3  0.1   0:00.12 top
     1 root      20   0  128104   6724   4180 S   0.0  0.2   0:12.60 systemd

I/O

09:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
09:10:01 AM     all      2.07      0.00      1.09      1.19      0.00     95.65
09:20:01 AM     all      2.09      0.00      1.01      1.10      0.00     95.79
09:30:01 AM     all      2.09      0.00      1.04      1.29      0.00     95.58
Average:        all      2.08      0.00      1.05      1.19      0.00     95.67
Manish Joisar
  • 1,256
  • 3
  • 23
  • 47
  • You should also check /var/opt/MarkLogic/Logs/ErrorLog.txt in each host in the cluster. It will also print out resource consumption information specific to the MarkLogic processes. – Mike Gardner Mar 29 '19 at 17:27
  • I suggest reaching out to MarkLogic Support, if you are entitled to.. – grtjn Apr 01 '19 at 08:41

1 Answers1

0

It looks like you expanded the security forest across additional nodes, and forest-security3 is not online.

Usually the Security database only resides on a single node in the cluster, with a replica on another node to provide a failover capability for logging in to administer the system.

The reason your forest is offline could be a resource contention issue (insufficient compute, memory and/or disk IO) issue, as that will sometimes cause timestamp lags. You can try restarting MarkLogic on all the nodes to see if that will allow it to recover from the error.

Mike Gardner
  • 6,611
  • 5
  • 24
  • 34
  • Thanks Michael for your reply. I have added resource contention information in my question, I restarted both servers, but still facing same issue. – Manish Joisar Mar 29 '19 at 09:38