8

I'm using scipy.optimize.minimize's COBYLA method to find a matrix of parameters for a categorical distribution. I need to impose the constraint that each parameter is greater than zero, and that the sum of the rows of the parameter matrix is a column of ones.

It's not clear to me how to implement this in scipy.minimize, because the constraints are checked for non-negativity rather than truth. The minimization raises an exception if I just pass the arrays as the constraint.

Does anyone know how to go about implementing these kinds of constraints?

ali_m
  • 71,714
  • 23
  • 223
  • 298
James Atwood
  • 4,289
  • 2
  • 17
  • 17
  • 1
    A member of my lab suggested a trick where the last column of parameters is fixed to one, and the parameters are given by the softmax of the output. This removes the need for the constraints that I mentioned. Still, I think an answer to this question would be useful to others. – James Atwood Feb 25 '16 at 15:36
  • This suggestions not really clear, can you give more details? – Stepan Yakovenko Jul 17 '19 at 10:28

2 Answers2

11

The first constraint x > 0 can be expressed very simply:

{'type':'ineq', 'fun': lambda x: x}

The second constraint is an equality constraint, which COBYLA doesn't natively support. However, you could express it as two separate inequality constraints instead:

{'type':'ineq', 'fun': lambda x: np.sum(x, 0) - 1}  # row sum >= 1
{'type':'ineq', 'fun': lambda x: 1 - np.sum(x, 0)}  # row sum <= 1

Otherwise you could try SLSQP instead, which does support equality constraints.

ali_m
  • 71,714
  • 23
  • 223
  • 298
  • This works, thanks! I have separate inequality constraint, so I'll stick with COBYLA. I've found that the upper/lower bound trick works pretty well. – James Atwood Feb 25 '16 at 15:45
  • Constraining only the smallest element rather than all of them is pretty smart. I'll have to remember that. – MB-F Feb 25 '16 at 15:50
  • 1
    It is smart but actually will make the model more difficult to solve as it introduces a discontinuity. I.e. in general it is not a good idea. Functions like min and max are to be avoided like the plague. – Erwin Kalvelagen Feb 25 '16 at 17:10
  • @ErwinKalvelagen You're right - I'll remove the mins. The reason I put them in there is because [scipy's COBYLA wrapper used to *only* accept scalar-valued constraint functions](https://github.com/scipy/scipy/pull/4648). It's been a long time since I've used that solver, and I didn't realize vector-valued constraints were now supported. – ali_m Feb 25 '16 at 21:39
8

You need equality constraints that enforce np.sum(x, 1) == 1 and inequality constraints for x >= 0.

However, the COBYLA method can only handle inequality constraints, as explained in the documentation of minimize (see the section that explains the constraints argument). Instead, you can use Sequential Least SQuares Programming (SLSQP), which supports both types of constraints. The minimize function should automatically choose the correct solver for you, based on the constraints you specify.

The constraints you need can be implemented like this:

def ineq_constraint(x):
    """constrain all elements of x to be >= 0"""
    return x

def eq_constraint(x):
    """constrain the sum of all rows to be equal to 1"""
    return np.sum(x, 1) - 1


constraints = [{'type': 'ineq', 'fun': ineq_constraint},
               {'type': 'eq', 'fun': eq_constraint}]

result = minimize(objective_function, x0, constraints=constraints)
MB-F
  • 22,770
  • 4
  • 61
  • 116
  • what if I have ''ineq_constraint'' returning a vector? How can I modify above? – Waqas Sep 25 '22 at 18:56
  • @Waqas `x` is a vector (actually numpy array)... no need to modify anything. (Unless the code/answer is outdated by now) – MB-F Sep 25 '22 at 20:48