2

I am currently trying to see if there are options to increase the overall memory limit of MarkLogic. I have attempted to increase the memory options in admin console but there seems to be no increase in it's limit. I looked through various forums and came across this thread. Stating that

"When the total hash join table size of all running SPARQL queries exceeds 50% of the host memory ... canceled with the "XDMP-MEMCANCELED" error"

So the question is can I increase the host memory % limit to say 90% for instance?

At the same time it also states that I can increase my memory to host to improve this. Do host memory refer to virtual memory? I tried increasing it but the memory limit when the error message is generated remains same.

System is currently running Windows.

===Update=== SPARQL Query Used

SELECT *
WHERE{
?people  </posted>  ?question .
?question </has_comment> ?comment .
?question </date_posted> ?question_date_posted .
?question </has_tags> ?tags .
?commented_by_person </commented_by> ?comment .
?comment </date_commented> ?comment_date_commented .
?comment </text> ?comment_text .
?people </from> ?school .
?people </has_age> ?age
}

Error Generated by Console

XDMP-MEMCANCELED: { SELECT people ... ?comment . } } -- Canceled because of memory usage on host desktop-60k8s55, requestMemory=4359540880, totalMemory=4359540880, memoryLimit=4294967296

Mike Gardner
  • 6,611
  • 5
  • 24
  • 34
WhiteSolstice
  • 641
  • 2
  • 7
  • 20
  • It's helpful to add additional context to your question, because it sounds like you have encountered an issue that you believe is memory related. That may well be true, there are also coding issues that will overtax memory. Are you seeing MEMCANCELED or other issues being logged by the server. – Mike Gardner Mar 28 '19 at 08:33
  • I have added the error message I hope it helps. Do let me know if it is not the correct message you are looking for. Right now I am currently trying to do lots of joins on purpose to see how much MarkLogic can handle and the time taken to execute these. Therefore, I am currently trying to search for ways to increase memory. @Michael Gardner – WhiteSolstice Mar 28 '19 at 09:12
  • I am currently running it on a local system @Michael Gardner – WhiteSolstice Mar 28 '19 at 09:15
  • Sorry but I am currently running it on a windows OS. I will definitely have a look at that if I ever touch a linux system. @Michael Gardner – WhiteSolstice Mar 28 '19 at 09:23

1 Answers1

3

MarkLogic memory consumption is a function of the various group and database cache settings, as well as the number of active forests, among other things.

I suggest checking out the Fundamentals of Resource Consumption paper, which has a section Memory Utilization in MarkLogic Server. It's also worthwhile to check out the Query and Tuning Guide

Host memory refers to the amount of RAM on the configured system or instance.

You can increase the amount of memory that MarkLogic will potentially consume by making changes to the group level settings.

In the Admin UI, go to the Configure -> Groups -> [Default] Configure tab, and ensure the 'cache sizing' is set to Manual and then you can adjust the various cache sizes as wanted.

You will also want to make sure that you have already done the appropriate OS level tuning for swap sizes and other best practices for the OS you are on.

If you later add memory to the system, then you will also need to make sure that you go back to adjust the Group caches and the OS swap.

Mike Gardner
  • 6,611
  • 5
  • 24
  • 34
  • As I forgot to confirm on one last thing from yestarday. I would like to ask if the default 50% host memory threshold on SPARQL join queries that will activate a "XDMP-MEMCANCELED" can be adjusted? @Michael Gardner – WhiteSolstice Mar 29 '19 at 08:33
  • 1
    You can expand the Triples caches, which are used by SPARQL. – Mike Gardner Mar 29 '19 at 17:08