3

I'm trying to solve the following problem: I have a lot (~80000) surface patches of an organ that's growing. I measure each of its areas over time (18 time-points) and want to fit a growth curve to it (bi-logistic model, eg. just the sum of two logistic functions bcs. there are two 'growth spurts' happening in the observed period).

I have box constraints to ensure that the exponential terms don't explode and a linear constraint that one growth spurt has to happen after the other. Also, in order to enforce some sort of spatial continuity in the fitted parameters, I add a penalty term to the objective function (least squares) given as the squared sum of differences between (some of the) parameters of neighbouring patches, so the individual model fits are not independent anymore.

I have gradients and the hessian of the whole thing, and I solve it at increasingly finer scales, starting with the overall surface area, subdividing, mapping the parameters of the global model fit to each patch, running the solver again, subdividing ... until I get my desired resolution.

So, optimization using IPOPT works, but it is terribly slow, and since I'm a bit of a noob in all things concerning optimization theory I was wondering if I'm doing something terribly stupid in the settings. I'm using ma86 as linear solver compiled against openBLAS running on a machine with 256G RAM and 56 cores as well as metis reordering. The other settings I use are:

% nlp scaling
solv_options.ipopt.nlp_scaling_method = 'gradient-based';
solv_options.ipopt.nlp_scaling_max_gradient = 1;
solv_options.ipopt.nlp_scaling_min_value = 1e-16;

solv_options.ipopt.bound_mult_init_method='constant';

% Barrier Parameter
solv_options.ipopt.mu_strategy = 'adaptive';
solv_options.ipopt.mu_oracle = 'quality-function';
solv_options.ipopt.fixed_mu_oracle = 'average_compl';
solv_options.ipopt.adaptive_mu_globalization = 'kkt-error';
solv_options.ipopt.corrector_type = 'affine';

% linear solver
solv_options.ipopt.max_soc=0;
solv_options.ipopt.accept_every_trial_step='yes';
solv_options.ipopt.linear_system_scaling = 'none';
solv_options.ipopt.neg_curv_test_tol = 0;
solv_options.ipopt.neg_curv_test_reg = 'yes';
solv_options.ipopt.max_refinement_steps=0;
solv_options.ipopt.min_refinement_steps=0;

% ma86 settings
solv_options.ipopt.linear_solver='ma86';
solv_options.ipopt.ma86_order='auto';
solv_options.ipopt.ma86_scaling='mc64';
solv_options.ipopt.ma86_small=1e-10;
solv_options.ipopt.ma86_static=1;
solv_options.ipopt.recalc_y='yes';

and this gives me something like this:

This is Ipopt version 3.12, running with linear solver ma86.

Number of nonzeros in equality constraint Jacobian...:        0
Number of nonzeros in inequality constraint Jacobian.:     2560
Number of nonzeros in Lagrangian Hessian.............:   112280

Total number of variables............................:     8960
                     variables with only lower bounds:     2560
                variables with lower and upper bounds:     3840
                     variables with only upper bounds:     2560
Total number of equality constraints.................:        0
Total number of inequality constraints...............:     1280
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:     1280

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  9.8260736e-01 0.00e+00 1.19e-02   0.0 0.00e+00    -  0.00e+00 0.00e+00   0
   1  9.6288666e-01 0.00e+00 1.24e-02  -4.5 1.14e-02   0.0 9.91e-01 1.00e+00f  1
   2  9.1582880e-01 0.00e+00 1.16e-02  -4.4 2.72e-02  -0.5 1.00e+00 1.00e+00f  1
   3  8.2635857e-01 0.00e+00 1.01e-02  -4.6 1.39e-01  -1.0 9.99e-01 1.00e+00f  1
   4  7.8943781e-01 0.00e+00 9.40e-03  -4.9 2.70e-02  -0.5 1.00e+00 1.00e+00f  1
   5  7.2123624e-01 0.00e+00 8.12e-03  -5.3 8.70e-02  -1.0 1.00e+00 1.00e+00f  1
   6  6.9535003e-01 0.00e+00 7.56e-03  -6.1 2.20e-02  -0.6 1.00e+00 9.06e-01f  1
   7  6.6635914e-01 0.00e+00 7.00e-03  -6.6 6.21e-02  -1.1 1.00e+00 5.40e-01f  1
   8  6.5683787e-01 0.00e+00 6.78e-03  -7.6 2.14e-02  -0.6 1.00e+00 3.81e-01f  1
   9  6.4238130e-01 0.00e+00 6.53e-03  -7.7 1.53e-01  -1.1 1.00e+00 2.90e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  10  6.3440625e-01 0.00e+00 6.33e-03  -8.7 2.20e-02  -0.7 1.00e+00 3.30e-01f  1
  11  6.2431009e-01 0.00e+00 6.16e-03  -8.3 3.08e-01  -1.2 1.00e+00 2.04e-01f  1
  12  6.1872460e-01 0.00e+00 6.05e-03  -8.8 2.29e-02  -0.7 1.00e+00 2.32e-01f  1
  13  6.0753815e-01 0.00e+00 5.86e-03  -8.9 2.73e-01  -1.2 1.00e+00 2.31e-01f  1
  14  6.0575477e-01 0.00e+00 5.82e-03  -9.9 2.60e-02  -0.8 1.00e+00 7.44e-02f  1
  15  6.0089103e-01 0.00e+00 5.71e-03 -11.0 1.11e-02  -0.4 1.00e+00 4.48e-01f  1
  16  5.9426852e-01 0.00e+00 5.58e-03 -11.0 3.14e-02  -0.8 1.00e+00 2.68e-01f  1
  17  5.9175418e-01 0.00e+00 5.52e-03 -11.0 1.17e-02  -0.4 1.00e+00 2.26e-01f  1
  18  5.8756155e-01 0.00e+00 5.44e-03 -11.0 3.97e-02  -0.9 1.00e+00 1.65e-01f  1
  19  5.8651702e-01 0.00e+00 5.42e-03  -9.5 1.26e-02  -0.5 1.00e+00 8.89e-02f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  20  5.8278151e-01 0.00e+00 5.34e-03 -10.3 5.18e-02  -0.9 1.00e+00 1.40e-01f  1
  21  5.8100614e-01 0.00e+00 5.30e-03 -11.0 1.37e-02  -0.5 1.00e+00 1.42e-01f  1
  22  5.7827149e-01 0.00e+00 5.25e-03 -11.0 7.12e-02  -1.0 1.00e+00 9.77e-02f  1
  23  5.7564914e-01 0.00e+00 5.19e-03  -9.4 1.52e-02  -0.6 1.00e+00 1.97e-01f  1
  24  5.7340095e-01 0.00e+00 5.15e-03 -10.0 1.06e-01  -1.0 1.00e+00 7.66e-02f  1
  25  5.7134734e-01 0.00e+00 5.10e-03 -11.0 1.78e-02  -0.6 1.00e+00 1.45e-01f  1
  26  5.6897098e-01 0.00e+00 5.06e-03 -11.0 1.82e-01  -1.1 1.00e+00 7.65e-02f  1
  27  5.6722484e-01 0.00e+00 5.02e-03 -11.0 2.11e-02  -0.7 1.00e+00 1.16e-01f  1
  28  5.6402838e-01 0.00e+00 5.25e-03 -11.0 3.29e-01  -1.1 1.00e+00 9.57e-02f  1
  29  5.6299118e-01 0.00e+00 5.26e-03 -10.5 2.38e-02  -0.7 1.00e+00 6.53e-02f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  30  5.5753178e-01 0.00e+00 5.63e-03 -11.0 1.38e-01  -1.2 1.00e+00 1.62e-01f  1
  31  5.5730217e-01 0.00e+00 5.63e-03 -11.0 2.37e-02  -0.8 1.00e+00 1.38e-02f  1
  32  5.5363398e-01 0.00e+00 5.71e-03 -11.0 8.55e-02  -1.2 1.00e+00 1.06e-01f  1
  33  5.5223091e-01 0.00e+00 5.70e-03 -11.0 2.43e-02  -0.8 1.00e+00 7.99e-02f  1
  34  5.5163441e-01 0.00e+00 5.70e-03  -9.9 1.30e-01  -1.3 1.00e+00 1.65e-02f  1
  35  5.4699694e-01 0.00e+00 5.67e-03 -10.9 2.56e-02  -0.9 1.00e+00 2.48e-01f  1
  36  5.4559780e-01 0.00e+00 5.67e-03 -11.0 2.30e-01  -1.3 1.00e+00 3.71e-02f  1
  37  5.4248376e-01 0.00e+00 5.63e-03 -11.0 2.58e-02  -0.9 1.00e+00 1.60e-01f  1
  38  5.3955718e-01 0.00e+00 5.63e-03 -10.7 6.78e-01  -1.4 1.00e+00 6.97e-02f  1
  39  5.3606672e-01 0.00e+00 5.57e-03 -10.9 2.79e-02  -1.0 1.00e+00 1.73e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  40  5.3481638e-01 0.00e+00 5.55e-03 -11.0 1.32e-02  -0.5 1.00e+00 1.33e-01f  1
  41  5.3248177e-01 0.00e+00 5.50e-03 -11.0 3.23e-02  -1.0 1.00e+00 1.10e-01f  1
  42  5.3143109e-01 0.00e+00 5.48e-03 -10.4 1.37e-02  -0.6 1.00e+00 1.05e-01f  1
  43  5.2975978e-01 0.00e+00 5.45e-03 -11.0 3.92e-02  -1.1 1.00e+00 7.50e-02f  1
  44  5.2855042e-01 0.00e+00 5.42e-03 -11.0 1.50e-02  -0.6 1.00e+00 1.13e-01f  1
  45  5.2670676e-01 0.00e+00 5.38e-03 -11.0 4.80e-02  -1.1 1.00e+00 7.85e-02f  1
  46  5.2428812e-01 0.00e+00 5.31e-03 -11.0 1.64e-02  -0.7 1.00e+00 2.12e-01f  1
  47  5.2221881e-01 0.00e+00 5.27e-03 -11.0 5.84e-02  -1.2 1.00e+00 8.45e-02f  1
  48  5.2084516e-01 0.00e+00 5.23e-03 -10.8 1.79e-02  -0.8 1.00e+00 1.14e-01f  1
  49  5.1928549e-01 0.00e+00 5.19e-03 -11.0 7.06e-02  -1.2 1.00e+00 6.06e-02f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  50  5.1787195e-01 0.00e+00 5.15e-03 -11.0 1.95e-02  -0.8 1.00e+00 1.10e-01f  1
  51  5.1639782e-01 0.00e+00 5.12e-03 -10.9 8.49e-02  -1.3 1.00e+00 5.45e-02f  1
  52  5.1606363e-01 0.00e+00 5.11e-03 -11.0 2.13e-02  -0.9 1.00e+00 2.45e-02f  1
  53  5.1178863e-01 0.00e+00 5.01e-03 -11.0 1.04e-01  -1.3 1.00e+00 1.50e-01f  1
  54  5.1173718e-01 0.00e+00 5.01e-03 -11.0 2.33e-02  -0.9 1.00e+00 3.61e-03f  1
  55  5.0944130e-01 0.00e+00 4.96e-03 -11.0 1.40e-01  -1.4 1.00e+00 7.70e-02f  1
  56  5.0727659e-01 0.00e+00 4.89e-03 -11.0 2.55e-02  -1.0 1.00e+00 1.43e-01f  1
  57  5.0638483e-01 0.00e+00 4.85e-03 -11.0 1.11e-02  -0.5 1.00e+00 1.29e-01f  1
  58  5.0426908e-01 0.00e+00 4.78e-03 -11.0 2.79e-02  -1.0 1.00e+00 1.32e-01f  1
  59  5.0333858e-01 0.00e+00 4.74e-03 -11.0 1.22e-02  -0.6 1.00e+00 1.26e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  60  5.0161958e-01 0.00e+00 4.69e-03 -11.0 3.05e-02  -1.1 1.00e+00 1.02e-01f  1
  61  5.0037437e-01 0.00e+00 4.64e-03 -11.0 1.34e-02  -0.6 1.00e+00 1.58e-01f  1
  62  4.9971029e-01 0.00e+00 4.62e-03 -11.0 3.34e-02  -1.1 1.00e+00 3.72e-02f  1
  63  4.9914235e-01 0.00e+00 4.59e-03 -11.0 1.48e-02  -0.7 1.00e+00 6.69e-02f  1
  64  4.9664256e-01 0.00e+00 4.51e-03 -11.0 3.70e-02  -1.2 1.00e+00 1.32e-01f  1
  65  4.9621567e-01 0.00e+00 4.50e-03 -11.0 1.62e-02  -0.7 1.00e+00 4.72e-02f  1
  66  4.9453149e-01 0.00e+00 4.44e-03 -11.0 4.19e-02  -1.2 1.00e+00 8.39e-02f  1
  67  4.9304626e-01 0.00e+00 4.38e-03 -11.0 1.79e-02  -0.8 1.00e+00 1.53e-01f  1
  68  4.9163087e-01 0.00e+00 4.34e-03 -11.0 6.89e-02  -1.3 1.00e+00 6.69e-02f  1
  69  4.9031174e-01 0.00e+00 4.28e-03 -11.0 1.97e-02  -0.8 1.00e+00 1.28e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  70  4.8795240e-01 0.00e+00 4.22e-03  -8.8 2.01e-01  -1.3 9.38e-01 1.04e-01f  1
  71  4.8721723e-01 0.00e+00 4.18e-03  -9.8 2.16e-02  -0.9 1.00e+00 6.73e-02f  1
  72  4.8560412e-01 0.00e+00 4.10e-03 -11.0 9.08e-03  -0.5 1.00e+00 3.33e-01f  1
  73  4.8395604e-01 0.00e+00 4.03e-03 -11.0 2.38e-02  -0.9 1.00e+00 1.42e-01f  1
  74  4.8308417e-01 0.00e+00 3.99e-03 -11.0 9.99e-03  -0.5 1.00e+00 1.69e-01f  1
  75  4.8236398e-01 0.00e+00 3.96e-03 -10.8 2.63e-02  -1.0 1.00e+00 5.83e-02f  1
  76  4.7919513e-01 0.00e+00 3.79e-03 -10.9 1.11e-02  -0.6 1.00e+00 5.72e-01f  1
  77  4.7844811e-01 0.00e+00 3.76e-03 -11.0 3.12e-02  -1.0 1.00e+00 5.79e-02f  1
  78  4.7801338e-01 0.00e+00 3.74e-03 -11.0 1.22e-02  -0.6 1.00e+00 7.40e-02f  1
  79  4.7631089e-01 0.00e+00 3.67e-03 -11.0 4.30e-02  -1.1 1.00e+00 1.23e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  80  4.7516941e-01 0.00e+00 3.61e-03 -11.0 1.35e-02  -0.7 1.00e+00 1.81e-01f  1
  81  4.7405503e-01 0.00e+00 3.57e-03 -11.0 7.24e-02  -1.1 1.00e+00 7.54e-02f  1
  82  4.7394863e-01 0.00e+00 3.57e-03 -11.0 1.49e-02  -0.7 1.00e+00 1.57e-02f  1
  83  4.7209146e-01 0.00e+00 3.51e-03  -8.9 1.76e-01  -1.2 9.65e-01 1.15e-01f  1
  84  4.7199185e-01 0.00e+00 3.50e-03 -10.1 1.66e-02  -0.8 1.00e+00 1.36e-02f  1
  85  4.7051045e-01 0.00e+00 3.41e-03 -11.0 6.70e-03  -0.3 1.00e+00 4.74e-01f  1
  86  4.7028360e-01 0.00e+00 3.40e-03 -11.0 1.92e-02  -0.8 1.00e+00 2.87e-02f  1
  87  4.6892068e-01 0.00e+00 3.31e-03 -11.0 7.44e-03  -0.4 1.00e+00 4.02e-01f  1
  88  4.6754658e-01 0.00e+00 3.25e-03 -11.0 2.37e-02  -0.9 1.00e+00 1.62e-01f  1
  89  4.6723856e-01 0.00e+00 3.23e-03 -11.0 8.24e-03  -0.4 1.00e+00 8.47e-02f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  90  4.6625331e-01 0.00e+00 3.18e-03 -11.0 3.07e-02  -0.9 1.00e+00 1.08e-01f  1
  91  4.6621348e-01 0.00e+00 3.18e-03 -11.0 9.16e-03  -0.5 1.00e+00 1.01e-02f  1
  92  4.6511327e-01 0.00e+00 3.17e-03 -11.0 4.12e-02  -1.0 1.00e+00 1.11e-01f  1
  93  4.6363418e-01 0.00e+00 3.14e-03 -11.0 1.02e-02  -0.5 1.00e+00 3.45e-01f  1
  94  4.6314545e-01 0.00e+00 3.14e-03 -11.0 6.17e-02  -1.0 1.00e+00 4.60e-02f  1
  95  4.6235630e-01 0.00e+00 3.12e-03 -11.0 1.13e-02  -0.6 1.00e+00 1.70e-01f  1
  96  4.6143809e-01 0.00e+00 3.11e-03  -9.7 1.03e-01  -1.1 1.00e+00 7.93e-02f  1
  97  4.6069624e-01 0.00e+00 3.10e-03 -10.6 1.38e-02  -0.6 1.00e+00 1.47e-01f  1
  98  4.5824023e-01 0.00e+00 3.68e-03  -8.6 3.14e-01  -1.1 8.52e-01 1.81e-01f  1
  99  4.5822224e-01 0.00e+00 3.68e-03  -9.9 2.27e-02  -0.7 1.00e+00 3.30e-03f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 100  4.5747364e-01 0.00e+00 3.71e-03 -11.0 7.25e-03  -0.3 1.00e+00 3.33e-01f  1
 101  4.5651156e-01 0.00e+00 3.77e-03 -11.0 2.65e-02  -0.8 1.00e+00 1.62e-01f  1
 102  4.5624508e-01 0.00e+00 3.79e-03 -11.0 8.36e-03  -0.3 1.00e+00 1.09e-01f  1
 103  4.5460694e-01 0.00e+00 3.91e-03 -10.5 3.10e-02  -0.8 1.00e+00 2.54e-01f  1
 104  4.5415701e-01 0.00e+00 3.92e-03 -11.0 9.64e-03  -0.4 1.00e+00 1.69e-01f  1
 105  4.5379967e-01 0.00e+00 3.95e-03 -11.0 3.54e-02  -0.9 1.00e+00 5.13e-02f  1
 106  4.5282822e-01 0.00e+00 3.99e-03 -11.0 1.10e-02  -0.4 1.00e+00 3.33e-01f  1
 107  4.5236345e-01 0.00e+00 4.03e-03 -11.0 4.03e-02  -0.9 1.00e+00 6.16e-02f  1
 108  4.5158337e-01 0.00e+00 4.06e-03 -11.0 1.24e-02  -0.5 1.00e+00 2.45e-01f  1
 109  4.4983066e-01 0.00e+00 4.19e-03  -8.9 4.54e-02  -1.0 1.00e+00 2.15e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 110  4.4976166e-01 0.00e+00 4.20e-03 -10.3 1.39e-02  -0.5 1.00e+00 2.01e-02f  1
 111  4.4733643e-01 0.00e+00 4.34e-03  -8.3 4.72e-02  -1.0 9.30e-01 2.78e-01f  1
 112  4.4674741e-01 0.00e+00 4.36e-03  -9.7 1.51e-02  -0.6 1.00e+00 1.60e-01f  1
 113  4.4600496e-01 0.00e+00 4.39e-03  -8.6 4.25e-02  -1.1 1.00e+00 8.06e-02f  1
 114  4.4596146e-01 0.00e+00 4.39e-03 -10.0 1.64e-02  -0.6 1.00e+00 1.09e-02f  1
 115  4.4157951e-01 0.00e+00 4.49e-03  -8.4 4.50e-02  -1.1 1.00e+00 4.49e-01f  1
 116  4.4137682e-01 0.00e+00 4.48e-03  -9.6 1.60e-02  -0.7 1.00e+00 4.90e-02f  1
 117  4.3873388e-01 0.00e+00 4.47e-03  -7.5 3.70e-02  -1.2 9.19e-01 2.63e-01f  1
 118  4.3815153e-01 0.00e+00 4.45e-03  -8.7 1.61e-02  -0.7 1.00e+00 1.33e-01f  1
 119  4.3569743e-01 0.00e+00 4.41e-03  -7.5 3.98e-02  -1.2 1.00e+00 2.34e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 120  4.3444019e-01 0.00e+00 4.36e-03  -8.6 1.67e-02  -0.8 1.00e+00 2.71e-01f  1
 121  4.3145745e-01 0.00e+00 4.27e-03  -7.3 4.24e-02  -1.3 1.00e+00 2.74e-01f  1
 122  4.2975412e-01 0.00e+00 4.19e-03  -8.0 1.81e-02  -0.8 1.00e+00 3.55e-01f  1
 123  4.2433966e-01 0.00e+00 3.98e-03  -6.9 4.56e-02  -1.3 1.00e+00 4.92e-01f  1
 124  4.2414838e-01 0.00e+00 3.96e-03  -7.5 1.92e-02  -0.9 1.00e+00 4.01e-02f  1
 125  4.2194620e-01 0.00e+00 3.87e-03  -6.9 4.80e-02  -1.4 1.00e+00 1.96e-01f  1
 126  4.1778190e-01 0.00e+00 3.63e-03  -7.7 2.08e-02  -0.9 1.00e+00 8.34e-01f  1
 127  4.1737428e-01 0.00e+00 2.94e-03  -7.1 4.72e-02  -1.4 1.00e+00 3.66e-02f  1
 128  4.1315312e-01 0.00e+00 2.54e-03  -7.8 2.22e-02  -1.0 1.00e+00 8.46e-01f  1
 129  4.1173197e-01 0.00e+00 2.51e-03  -7.1 6.31e-02  -1.5 1.00e+00 1.27e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 130  4.0915489e-01 0.00e+00 2.46e-03  -7.7 2.32e-02  -1.0 1.00e+00 5.13e-01f  1
 131  4.0515277e-01 0.00e+00 2.38e-03  -6.9 9.97e-02  -1.5 1.00e+00 3.50e-01f  1
 132  4.0343660e-01 0.00e+00 2.34e-03  -7.3 2.38e-02  -1.1 1.00e+00 3.43e-01f  1
 133  3.9516682e-01 0.00e+00 2.17e-03  -6.8 1.68e-01  -1.6 1.00e+00 7.24e-01f  1
 134  3.9098427e-01 0.00e+00 2.06e-03  -7.2 2.34e-02  -1.1 1.00e+00 9.07e-01f  1
 135  3.9058959e-01 0.00e+00 2.05e-03  -7.0 5.50e-02  -1.6 1.00e+00 3.97e-02f  1
 136  3.8628795e-01 0.00e+00 1.93e-03  -7.7 2.39e-02  -1.2 1.00e+00 9.42e-01f  1
 137  3.8288703e-01 0.00e+00 1.84e-03  -7.2 6.76e-02  -1.7 1.00e+00 3.49e-01f  1
 138  3.8113960e-01 0.00e+00 1.79e-03  -7.8 2.37e-02  -1.2 1.00e+00 4.02e-01f  1
 139  3.7895865e-01 0.00e+00 1.74e-03  -7.3 8.98e-02  -1.7 1.00e+00 2.23e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 140  3.7630588e-01 0.00e+00 1.65e-03  -7.9 2.56e-02  -1.3 1.00e+00 6.04e-01f  1
 141  3.7040736e-01 0.00e+00 2.44e-03  -7.4 4.40e-01  -1.8 1.00e+00 5.52e-01f  1
 142  3.6744301e-01 0.00e+00 2.20e-03  -8.1 5.47e-02  -1.3 1.00e+00 6.87e-01f  1
 143  3.6668338e-01 0.00e+00 2.11e-03  -9.7 1.27e-02  -0.9 1.00e+00 4.33e-01f  1
 144  3.6563026e-01 0.00e+00 2.04e-03 -10.0 1.45e-01  -1.4 1.00e+00 2.33e-01f  1
 145  3.6473497e-01 0.00e+00 1.94e-03 -11.0 1.73e-02  -1.0 1.00e+00 4.85e-01f  1
 146  3.6423323e-01 0.00e+00 2.14e-03 -11.0 2.77e-01  -1.5 1.00e+00 1.00e-01f  1
 147  3.6397151e-01 0.00e+00 2.15e-03 -10.1 2.02e-02  -1.0 1.00e+00 1.32e-01f  1
 148  3.6196968e-01 0.00e+00 2.50e-03  -8.5 1.56e-01  -1.5 1.00e+00 3.92e-01f  1
 149  3.6176499e-01 0.00e+00 2.49e-03  -9.6 1.65e-02  -1.1 1.00e+00 1.00e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 150  3.6069639e-01 0.00e+00 2.45e-03  -8.4 3.49e-02  -1.6 1.00e+00 2.18e-01f  1
 151  3.6041434e-01 0.00e+00 2.43e-03  -9.4 1.63e-02  -1.1 1.00e+00 1.30e-01f  1
 152  3.5921303e-01 0.00e+00 2.37e-03  -8.5 3.78e-02  -1.6 1.00e+00 2.34e-01f  1
 153  3.5869485e-01 0.00e+00 2.33e-03  -9.4 1.64e-02  -1.2 1.00e+00 2.27e-01f  1
 154  3.5699552e-01 0.00e+00 2.25e-03  -7.9 4.06e-02  -1.7 1.00e+00 3.20e-01f  1
 155  3.5674149e-01 0.00e+00 2.23e-03  -8.7 1.77e-02  -1.2 1.00e+00 1.07e-01f  1
 156  3.5471902e-01 0.00e+00 2.11e-03  -7.5 4.33e-02  -1.7 1.00e+00 3.71e-01f  1
 157  3.5335745e-01 0.00e+00 1.99e-03  -8.3 1.91e-02  -1.3 1.00e+00 5.64e-01f  1
 158  3.5245420e-01 0.00e+00 1.94e-03  -7.5 4.53e-02  -1.8 1.00e+00 1.67e-01f  1
 159  3.5021073e-01 0.00e+00 1.73e-03  -8.3 2.05e-02  -1.3 1.00e+00 9.16e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 160  3.4610138e-01 0.00e+00 1.04e-03  -7.3 4.72e-02  -1.8 1.00e+00 7.94e-01f  1
 161  3.4469412e-01 0.00e+00 9.99e-04  -8.0 2.07e-02  -1.4 1.00e+00 6.40e-01f  1
 162  3.4399767e-01 0.00e+00 1.02e-03  -7.7 4.72e-02  -1.9 1.00e+00 1.42e-01f  1
 163  3.4260897e-01 0.00e+00 9.60e-04  -8.1 2.22e-02  -1.4 1.00e+00 6.21e-01f  1
 164  3.4129508e-01 0.00e+00 9.37e-04  -7.4 4.92e-02  -1.9 1.00e+00 2.68e-01f  1
 165  3.4016827e-01 0.00e+00 9.15e-04  -8.0 2.34e-02  -1.5 1.00e+00 5.07e-01f  1
 166  3.3825082e-01 0.00e+00 8.81e-04  -7.2 5.10e-02  -2.0 1.00e+00 3.98e-01f  1
 167  3.3610567e-01 0.00e+00 8.39e-04  -7.7 2.44e-02  -1.5 1.00e+00 1.00e+00f  1
 168  3.3460677e-01 0.00e+00 8.12e-04  -7.1 5.92e-02  -2.0 1.00e+00 3.31e-01f  1
 169  3.3266040e-01 0.00e+00 7.74e-04  -7.7 2.51e-02  -1.6 1.00e+00 9.50e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 170  3.3037125e-01 0.00e+00 7.34e-04  -7.1 8.30e-02  -2.1 1.00e+00 5.31e-01f  1
 171  3.2873890e-01 0.00e+00 7.02e-04  -7.7 2.53e-02  -1.6 1.00e+00 8.56e-01f  1
 172  3.2622239e-01 0.00e+00 6.57e-04  -7.2 5.31e-02  -2.1 1.00e+00 6.32e-01f  1
 173  3.2470308e-01 0.00e+00 6.28e-04  -7.8 2.53e-02  -1.7 1.00e+00 8.69e-01f  1
 174  3.2372187e-01 0.00e+00 6.27e-04  -7.5 5.26e-02  -2.2 1.00e+00 2.63e-01f  1
 175  3.2283458e-01 0.00e+00 5.92e-04  -8.2 2.60e-02  -1.7 1.00e+00 5.20e-01f  1
 176  3.2183252e-01 0.00e+00 6.17e-04  -7.5 5.47e-02  -2.2 1.00e+00 2.71e-01f  1
 177  3.2046827e-01 0.00e+00 5.47e-04  -8.1 2.70e-02  -1.8 1.00e+00 8.08e-01f  1
 178  3.1975003e-01 0.00e+00 6.03e-04  -7.1 5.85e-02  -2.3 1.00e+00 2.04e-01f  1
 179  3.1900035e-01 0.00e+00 5.19e-04  -7.6 2.77e-02  -1.8 1.00e+00 4.53e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 180  3.1615561e-01 0.00e+00 4.90e-04  -6.8 6.32e-02  -2.3 1.00e+00 8.53e-01f  1
 181  3.1524761e-01 0.00e+00 4.49e-04  -7.3 2.68e-02  -1.9 1.00e+00 5.85e-01f  1
 182  3.1326122e-01 0.00e+00 4.87e-04  -7.0 6.24e-02  -2.4 9.98e-01 6.26e-01f  1
 183  3.1296679e-01 0.00e+00 4.09e-04  -7.7 2.63e-02  -1.9 1.00e+00 2.01e-01f  1
 184  3.1055716e-01 0.00e+00 4.34e-04  -6.8 6.57e-02  -2.4 1.00e+00 8.22e-01f  1
 185  3.1020674e-01 0.00e+00 3.60e-04  -7.5 2.70e-02  -2.0 1.00e+00 2.54e-01f  1
 186  3.0929569e-01 0.00e+00 4.68e-04  -7.0 6.86e-02  -2.5 1.00e+00 3.18e-01f  1
 187  3.0819030e-01 0.00e+00 3.23e-04  -7.8 2.94e-02  -2.0 1.00e+00 8.02e-01f  1
 188  3.0763044e-01 0.00e+00 4.47e-04  -7.2 8.21e-02  -2.5 1.00e+00 2.02e-01f  1
 189  3.0725472e-01 0.00e+00 3.07e-04  -8.0 3.11e-02  -2.1 1.00e+00 2.85e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 190  3.0578606e-01 0.00e+00 4.05e-04  -7.5 2.33e-01  -2.6 1.00e+00 5.11e-01f  1
 191  3.0495222e-01 0.00e+00 3.43e-04  -8.1 2.91e-02  -2.2 1.00e+00 6.59e-01f  1
 192  3.0413633e-01 0.00e+00 1.17e-03  -7.6 4.93e-01  -2.6 1.00e+00 2.84e-01f  1
 193  3.0331078e-01 0.00e+00 9.60e-04  -7.7 3.14e-02  -2.2 1.00e+00 6.81e-01f  1
 194  3.0239991e-01 0.00e+00 8.65e-04  -6.7 7.67e-02  -2.7 9.99e-01 4.00e-01f  1
 195  3.0137954e-01 0.00e+00 6.77e-04  -7.5 3.36e-02  -2.3 1.00e+00 8.52e-01f  1
 196  3.0056119e-01 0.00e+00 6.24e-04  -7.5 8.05e-02  -2.7 1.00e+00 3.28e-01f  1
 197  3.0014178e-01 0.00e+00 5.65e-04  -8.3 3.58e-02  -2.3 1.00e+00 3.62e-01f  1
 198  2.9923633e-01 0.00e+00 5.17e-04  -8.2 8.53e-02  -2.8 1.00e+00 3.59e-01f  1
 199  2.9888005e-01 0.00e+00 4.75e-04  -8.9 3.83e-02  -2.4 1.00e+00 3.15e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 200  2.9848396e-01 0.00e+00 4.57e-04  -8.1 8.98e-02  -2.8 1.00e+00 1.58e-01f  1
 201  2.9821853e-01 0.00e+00 4.30e-04  -8.8 4.15e-02  -2.4 1.00e+00 2.29e-01f  1
 202  2.9738245e-01 0.00e+00 3.96e-04  -7.2 9.62e-02  -2.9 9.99e-01 3.39e-01f  1
 203  2.9665614e-01 0.00e+00 3.36e-04  -7.8 4.41e-02  -2.5 1.00e+00 6.30e-01f  1
 204  2.9578199e-01 0.00e+00 3.07e-04  -6.8 1.02e-01  -2.9 1.00e+00 3.87e-01f  1
 205  2.9465429e-01 0.00e+00 2.39e-04  -7.4 4.54e-02  -2.5 1.00e+00 1.00e+00f  1
 206  2.9301415e-01 0.00e+00 1.99e-04  -6.7 1.07e-01  -3.0 1.00e+00 8.01e-01f  1
 207  2.9292261e-01 0.00e+00 1.94e-04  -7.3 4.47e-02  -2.6 1.00e+00 8.17e-02f  1
 208  2.9132376e-01 0.00e+00 2.05e-04  -6.8 1.11e-01  -3.0 1.00e+00 7.08e-01f  1
 209  2.9068030e-01 0.00e+00 1.46e-04  -7.4 4.72e-02  -2.6 1.00e+00 5.83e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 210  2.8989341e-01 0.00e+00 2.15e-04  -6.8 1.19e-01  -3.1 1.00e+00 3.71e-01f  1
 211  2.8905846e-01 0.00e+00 1.25e-04  -7.4 5.00e-02  -2.7 1.00e+00 7.99e-01f  1
 212  2.8817010e-01 0.00e+00 2.35e-04  -6.9 1.32e-01  -3.1 1.00e+00 4.36e-01f  1
 213  2.8741077e-01 0.00e+00 1.25e-04  -7.5 5.21e-02  -2.7 1.00e+00 7.41e-01f  1
 214  2.8705020e-01 0.00e+00 2.96e-04  -7.2 1.57e-01  -3.2 1.00e+00 1.65e-01f  1
 215  2.8639044e-01 0.00e+00 1.29e-04  -8.0 5.54e-02  -2.8 1.00e+00 6.44e-01f  1
 216  2.8567025e-01 0.00e+00 2.13e-04  -7.4 4.30e-01  -3.2 1.00e+00 3.16e-01f  1
 217  2.8519078e-01 0.00e+00 1.22e-04  -8.1 6.05e-02  -2.8 1.00e+00 4.79e-01f  1
 218  2.8508336e-01 0.00e+00 1.93e-04  -7.5 3.80e-01  -3.3 1.00e+00 4.58e-02f  1
 219  2.8459190e-01 0.00e+00 1.16e-04  -8.3 6.70e-02  -2.9 1.00e+00 4.75e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
 220  2.8382163e-01 0.00e+00 1.63e-04  -7.2 3.30e-01  -3.3 1.00e+00 3.29e-01f  1
 221  2.8341130e-01 0.00e+00 1.28e-04  -7.8 7.27e-02  -2.9 1.00e+00 3.93e-01f  1
 222  2.8284739e-01 0.00e+00 1.56e-04  -7.0 2.21e-01  -3.4 1.00e+00 2.45e-01f  1
 223  2.8183563e-01 0.00e+00 1.19e-04  -7.7 7.96e-02  -3.0 1.00e+00 9.45e-01f  1
 224  2.8117611e-01 0.00e+00 1.40e-04  -6.9 4.77e-01  -3.4 1.00e+00 2.96e-01f  1
 225  2.8033847e-01 0.00e+00 1.04e-04  -7.6 8.59e-02  -3.0 1.00e+00 7.96e-01f  1
 226  2.8005033e-01 0.00e+00 9.73e-05  -9.1 3.26e-02  -2.6 1.00e+00 6.19e-01f  1

Number of Iterations....: 226

                                   (scaled)                 (unscaled)
Objective...............:   2.8005032523583379e-01    2.8005032523583379e-01
Dual infeasibility......:   9.7272674831558608e-05    9.7272674831558608e-05
Constraint violation....:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   3.2315149852880829e-07    3.2315149852880829e-07
Overall NLP error.......:   9.7272674831558608e-05    9.7272674831558608e-05


Number of objective function evaluations             = 227
Number of objective gradient evaluations             = 227
Number of equality constraint evaluations            = 0
Number of inequality constraint evaluations          = 228
Number of equality constraint Jacobian evaluations   = 0
Number of inequality constraint Jacobian evaluations = 1
Number of Lagrangian Hessian evaluations             = 226
Total CPU secs in IPOPT (w/o function evaluations)   =   3403.326
Total CPU secs in NLP function evaluations           =    214.977

EXIT: Optimal Solution Found.

.... the timing is ok at this resolution, but at higher ones, the linear solves become quite large and extremely slow. So I guess my question is: can I expect something like this (at a higher resolution, with ~500 000 parameters) to take a couple of days to solve or are there any settings I could tweak to increase convergence / reduce the number if iterations needed. The initial guesses are quite good actually btw, and a local optimum is actually all I'm looking for.

mrburnst
  • 85
  • 1
  • 5
  • I usually try out different NLP solvers. There can be quite a variation in performance between different solvers. Sometimes details in the formulation of the least squares problem can also make a big difference. May be you can find a good starting point using fewer observations, and then refine with all data (or otherwise starting smaller). – Erwin Kalvelagen Jul 13 '18 at 00:26
  • thanks, yes, I noticed. I only tried fmincon's interior point implementation and IPOPT as I need box as well as linear inequality and equality constraints and that's what I have at hand. I'm pretty confident about my starting points, I actually do optimise at increasingly finer resolutions and use previous (converged) results as initialisation. My main question was about what runtime to expect from an IP-algorithm on a problem of that size (~500k variables, 100k constraints) and if there are alternatives to IP methods that can handle that type of problem and might converge more quickly. – mrburnst Jul 13 '18 at 07:10

2 Answers2

3

It looks like you were very thorough in your approach. With ~500k variables and 100k constraints you have a lot of degrees of freedom. The primary alternative to interior point approaches (of which, IPOPT is quite good) is an active-set approach. Active-set methods tend to be better with few degrees of freedom, so IPOPT is your best bet.

The IPOPT output indicates a few things:

  1. The Hessian is not positive-definite (nearly every iteration requires regularization)

  2. After regularization, the problem is sufficiently convex within variable bounds (no back-tracking line search)

  3. IPOPT time is >> function call time (3403.326 : 214.977 is very large). Most of this time is in matrix factorizations.

  4. Nearly every iteration is truncated because of variable bounds.

It is my understanding that when your problem gets too big it exceeds hardware limitations (CPU cache) and the linear algebra time blows up. This is probably your main problem.

For these reasons, I would recommend trying a BFGS approach (IPOPT option). By directly calculating the approximate inverse of the Hessian, you avoid difficult matrix factorizations/solves. Further, the BFGS approach can guarantee positive-definite Hessians. Usually, the BFGS approximation is used when the Hessian is not available since the exact Hessian should provide more accurate steps. But with your regularization, expensive factorization and truncated steps, BFGS will likely be almost as good. Expect more iterations (226 is very small), but each should be much faster.

You may also want to play with loosening your variable bounds. Interior-point methods are designed to avoid going to bounds. With that many bounds, it could be slowing down progress.

makunha
  • 411
  • 3
  • 6
  • hi! thanks so much for this insightful answer. I already tried without the Hessian, and the results I got were much worse. So, I guess the projection unto the semi-definite cone IPOPT is doing is 'better' than the approximation from the first derivatives? My other idea would have been to replace my constraints by penalty terms in the objective function. Then the constraints will of course not be fulfilled anymore (only penalised), but I would end up with a much simpler problem. What do you recon? – mrburnst Jul 15 '18 at 19:31
  • What do you mean by "much worse"? Was it slower, infeasible or did it find a worse local optima? – makunha Jul 16 '18 at 18:38
  • Nothing about exact second derivatives vs BFGS guarantees a "better" local minima -- that's just luck and initial conditions. You said you were just looking for a local optima, but if you want more global optima you can borrow ideas from the global optimization community. Things like spanning the space of feasible initial conditions could work, especially with BFGS since it's faster. – makunha Jul 17 '18 at 17:42
0

Not a definite answer but too long for a comment: Ipopt should be well optimized already so I'm afraid you won't be getting better results unless you change your whole algorithm.

If a local optimum is what you're looking for, then start at a coarse resolution and choose the "most promising" section (i.e. where an optimum is more likely to be found). Then enhance your resolution on that section and start over by dichotomy.

You could also check Neural Networks, which can model complex non-linear functions and are now mature, with plenty of implementation taking advantage of CPU+GPU architectures that could help a lot. The gradient backprop is designed to efficiently find local extremas.

Matthieu
  • 2,736
  • 4
  • 57
  • 87
  • thanks for your suggestion. NN won't be the solution here, I'm afraid. The issue is not that the problem is too nonlinear - I know the analytical gradient and hessian of what I'm trying to solve. just the large dimensionality makes it hard to optimize numerically. – mrburnst Jul 11 '18 at 04:09