Mosek memory issue for large linear programming

Question

I use MOSEK to run a very large linear programming problem in Matlab (32768 unknowns and 691621 constraints). The code is submitted in a Linux based cluster. In the bash file I request the following amount of memory:

#$ -l h_vmem=20G
#$ -l tmem=20G

but get Mosek error: MSK_RES_ERR_SPACE (Out of space.)

I could request more memory (however, it is unclear how much more?), but this would mean queuing in the cluster for a long time.

Hence, I was wondering whether I can try to ameliorate the issue in some other way.

1) Quoting from some MOSEK FAQs:

Java, .NET, amd Python applications runs under a virtual machine. MOSEK shares memeory
with the virtual machine. This implies it might be necessary to force the virtual machine to
free unused memory by explicitly calling the garbage collector (for example before optimization
is performed) in order to make sufficient memory available to MOSEK.

Can this advise be useful? What does it mean calling the garbage collector (i.e., which line should I add to my Matlab code?).

2) From https://docs.mosek.com/9.2/pythonapi/guidelines-optimizer.html (even though this is for Python), it suggests to set

Task.putmaxnumvar. Estimate for the number of variables.
Task.putmaxnumcon. Estimate for the number of constraints.
Task.putmaxnumcone. Estimate for the number of cones.
Task.putmaxnumbarvar. Estimate for the number of semidefinite matrix variables.
Task.putmaxnumanz. Estimate for the number of non-zeros in A.
Task.putmaxnumqnz. Estimate for the number of non-zeros in the quadratic terms.

Can I do that in Matlab? How?

3) From http://ask.cvxr.com/t/how-to-deal-with-out-of-space-error-when-using-mosek-to-solve-a-conic-optimization-problem/7510: "It will reduce memory consumption to some extent if you run on 1 thread (set MSK_IPAR_NUM_THREADS to 1 in cvx solver options or set MSK_IPAR_INTPNT_MULTI_THREAD to 0)"

Can this be done in Matlab as well? I have tried

param_MOSEK.MSK_IPAR_NUM_THREADS = 1;
param_MOSEK.MSK_IPAR_INTPNT_MULTI_THREAD = 'MSK_OFF';

but it does not seem to work as the output file still gives

Optimizer  - threads                : 16              
Optimizer  - solved problem         : the dual        
...

---

Comments related to questions below:

• The code runs in my MacOS 64 bit using 16 threads in just 180 sec. The memory of the computer is 32 GB 2667 MHz DDR4. It uses much less than 20G (around 9G).

• The code fails when it is run on the cluster of my univ (Linux based) after having requested 20G of vmem and tmem. In the cluster, MOSEK executes the presolving, the GP based matrix reordering, and then fails. This is a typical log file

Wed 9 Sep 08:10:47 BST 2020 Task ID is 6

                            < M A T L A B (R) >
                  Copyright 1984-2019 The MathWorks, Inc.
              R2019b Update 3 (9.7.0.1261785) 64-bit (glnxa64)
                             November 27, 2019
 
For online documentation, see https://www.mathworks.com/support
For product information, visit www.mathworks.com.
 

MOSEK Version 9.2.5 (Build date: 2020-4-22 22:56:56)
Copyright (c) MOSEK ApS, Denmark. WWW: mosek.com
Platform: Linux/64-X86

Problem
  Name                   :                 
  Objective sense        : min             
  Type                   : LO (linear optimization problem)
  Constraints            : 691597          
  Cones                  : 0               
  Scalar variables       : 32768           
  Matrix variables       : 0               
  Integer variables      : 0               

Optimizer started.
Presolve started.
Linear dependency checker started.
Linear dependency checker terminated.
Eliminator started.
Freed constraints in eliminator : 0
Eliminator terminated.
Eliminator - tries                  : 1                 time                   : 0.00            
Lin. dep.  - tries                  : 1                 time                   : 0.33            
Lin. dep.  - number                 : 0               
Presolve terminated. Time: 2.99    
GP based matrix reordering started.
GP based matrix reordering terminated.
Optimizer terminated. Time: 20.15   


Interior-point solution summary
  Problem status  : UNKNOWN
  Solution status : UNKNOWN
  Primal.  obj: 0.0000000000e+00    nrm: 1e+00    Viol.  con: 1e+00    var: 0e+00  
  Dual.    obj: 0.0000000000e+00    nrm: 0e+00    Viol.  con: 0e+00    var: 0e+00  
Optimizer summary
  Optimizer                 -                        time: 20.15   
    Interior-point          - iterations : 0         time: 19.95   
      Basis identification  -                        time: 0.00    
        Primal              - iterations : 0         time: 0.00    
        Dual                - iterations : 0         time: 0.00    
        Clean primal        - iterations : 0         time: 0.00    
        Clean dual          - iterations : 0         time: 0.00    
    Simplex                 -                        time: 0.00    
      Primal simplex        - iterations : 0         time: 0.00    
      Dual simplex          - iterations : 0         time: 0.00    
    Mixed integer           - relaxations: 0         time: 0.00    

Mosek error: MSK_RES_ERR_SPACE (Out of space.)

Answer 1

1. Irrelevant in Matlab.

2. Irrelevant and imposible in Matlab. The MEX interface feeds the problem into Mosek in one go and takes care of all allocations itself.

3. For MSK_IPAR_NUM_THREADS to be respected you must restart the whole process i.e. Matlab. See https://docs.mosek.com/9.2/faq/faq.html#mosek-is-ignoring-the-limit-on-the-number-of-threads. However, when you set MSK_IPAR_INTPNT_MULTI_THREAD = 'MSK_OFF' then Mosek will use 1 thread regardless of the number of all threads available i.e. the number printed to the log is just an upper bound. You should be able to see in the task manager/top/whatever other CPU load tracker that only 1CPU is in use.

The basic question is: have you tried to run the problem without any memory limits to see if it works at all and estimate the memory consumption? Does it run on other machines?