SGE enables users to set limits on virtual memory/vmem usage (e.g. the h_vmem argument for a job submission).
But how exactly does SGE monitor VMEM usage and send a kill signal if it is exceeded? Does it poll at some frequency? Add up some kernel provided value across a process tree? How does this work mechanistically? Even an incomplete explanation or simple pointer to source code would be greatly appreciated.
I am not familiar with how SGE works in detail, simply used to administer a small cluster that used it a while ago. However, what you ask reminds me of the following script I use often to report memory of a process:
https://github.com/jhclark/memusg
Basically, a command run within the qsub script is a child process of this script and/or the SGE monitor (qmon). Therefore, there is likely a method somewhere that monitors memory usage insimilar manner as the Python code linked above. The relevant section in the code is:
proc = Popen(child_command, stdin=None, stdout=None, stderr=None, env=None, shell=True)
vmpeak = -1
while proc.returncode == None:
vmpeak = max(get_vsize(sid), vmpeak)
log("Waiting for child to exit. vmpeak={}".format(vmpeak))
proc.poll()
sleep(0.1) # Time in seconds (float)
out.write("memusg: vmpeak: {} kb\n".format(vmpeak))
Where child_command is the actual command we want to run. The code starts a process using this command and monitors it at regular intervals, and in this case reporting the max memory when the process completes. It would be trivial to do change this code to break out of the loop and kill the child process if memory exceeds some maximum.
Hope this helps.
