1

I want to be able to set up memory requirements per job.

For instance: to run 5 jobs, for which I know each will need 4 GB memory. I have 16 GB RAM on the Ubuntu server and 16 GB swap. I want to avoid using the swap. Can I do something like:

qsub -l mem_required_for_my_job=4G job1
qsub -l mem_required_for_my_job=4G job2
qsub -l mem_required_for_my_job=4G job3
qsub -l mem_required_for_my_job=4G job4
qsub -l mem_required_for_my_job=4G job5

? The jobs will require 4 GB at some moment, but not in the beginning.

How to tell SGE what my requirements are? How to avoid scheduling 5 x 4 GB when only 16 GB available?

I read the user guide and tried s_vmem, h_vmem, mem_free, mem_used. None of them is what I want. I do not want my jobs to get killed in the middle of the processing. I want them not to be scheduled, unless the maximum resources needed are available.

Can I do this?

Thank you all!

Pavel
  • 1,038
  • 1
  • 11
  • 30
grs
  • 2,235
  • 6
  • 28
  • 36

2 Answers2

2

On our cluster we use h_vmem to enforce job memory allocation.

The thing you appear to be missing is setting the available amount as a consumable resource. In qconf -mc or in the qmon complexes dialog you need to set the resource as requestable and consumable.

Then, on each host with qconf -me you need to set the amount of available memory in complex_values.

For example we have host definitions that look like:

hostname              node004
load_scaling          NONE
complex_values        h_vmem=30.9G,exclusive=1,version=3.3.0
Kamil Kisiel
  • 12,184
  • 7
  • 48
  • 69
  • I tried with `h_vmem`, `virtual_free` and quotas. None of them carry the task in the way I want it. `h_vmem` protects the system, killing the jobs if they overcome their requirements. Both `quota` and `virtual_free` didn't do anything to prevent the memory exhausting. The OS killed the job in this case. I want to avoid killing the jobs. I would like to make them stay in the queue, waiting for their requested resources to be available. Is this possible? – grs Mar 02 '11 at 22:28
  • I'm not sure what you mean. With the `h_vmem` limit, GE will only run the job on the node if it has enough of the `h_vmem` resource available to run it. It also sets the ulimit on virtual memory. If the job tries to allocate more than the limit, it will be killed by the OS. – Kamil Kisiel Mar 03 '11 at 01:06
  • Exactly. I want to be able to allocate memory requirements per job, but do not want to kill the job if exceeds these requirements (for now). What I can't understand is how GE allocates memory. If I have 5 jobs x 4GB each in their peak, how many would run on 16 GB simultaneously, without swapping? If 4 jobs run and takes total of 10GB in their beginning, would the 5th one go in? How GE will know what is the expected memory usage peak per job? – grs Mar 03 '11 at 01:29
  • 1
    GridEngine will run exactly as many jobs as match the available complex resources. So if you define a node to have 16GB of h_vmem available, and you submit 5 jobs requesting 4GB, GridEngine will only place 4 of them on the node at once. Your job should always request the peak amount it expects to use. If you have swap space on your nodes, you can include that amount in the h_vmem complex value. – Kamil Kisiel Mar 03 '11 at 02:22
  • Great! What arguments should I use to specify my job requirements? I believe it should be something like `qsub -l mem_required=4G job1`. What I have to use after the `-l` part? – grs Mar 03 '11 at 05:51
  • 1
    The arguments are the same as the name of the complex. So if you have `h_vmem` configured as a consumable resource and under a host's complex_values, your command would look like: `qsub -l h_vmem=4G job1`. – Kamil Kisiel Mar 03 '11 at 06:21
  • 2
    Also here's a quick blog post from gridengine.info regarding memory limits: http://gridengine.info/2009/12/01/adding-memory-requirement-awareness-to-the-scheduler – Kamil Kisiel Mar 03 '11 at 06:22
  • Just to complete the discussion: if I setup `s_vmem` and `h_vmem` in my complex resources via `qconf -me hostname`, then I must pass both of them: `qsub -l s_vmem=2G,h_vmem=3G job1`. If it is just one or none, the job just quits without any sign why. So `-l ,` become mandatory for everyone. Thanks! – grs Mar 03 '11 at 22:06
0

I think the what you are looking for is virtual_free

"If a job request for virtual_free exceeds the available amount, the job is not dispatched to a queue on that host." from https://web.archive.org/web/20100801220839/http://wikis.sun.com/display/GridEngine/Configuring+Complex+Resource+Attributes

See also http://markmail.org/message/2iw4esthqvs5rc2a

Community
  • 1
rbtgde
  • 1