Why am I getting this DCPError?

Question

I'm trying to optimize a binary portfolio vector to be greater than a benchmark using CVXPY.

import cvxpy as cp
import numpy as np

# Generate a random non-trivial quadratic program.

n = 10 # number of options

np.random.seed(1)
mu = np.random.randn(n) # expected means
var_covar = np.random.randn(n,n) # variance-covariance matrix
var_covar = var_covar.T.dot(var_covar) # cont'd
bench_cov = np.random.randn(n) # n-length vector of cov(benchmark, returns)

lamd = 0.01 # risk tolerance

# Define and solve the CVXPY problem.

x = cp.Variable(n, boolean=True)

prob = cp.Problem(cp.Maximize(mu.T@x + lamd * (cp.quad_form(x, var_covar) - (2 * bench_cov.T@x))), [cp.sum(x) == 4])

prob.solve()

I get this error using CVXPY version 1.1.0a0 (downloaded directly from github):

DCPError: Problem does not follow DCP rules. Specifically:

The objective is not DCP, even though each sub-expression is.

You are trying to maximize a function that is convex.

From what I've read maximizing a convex function is very difficult, but I got this equation from a paper. I figure I must be doing something wrong as I'm new to quadratic programming and CVXPY.

Thank you!

Maximizing a convex function is a *non*-convex problem. Hard to solve non-convex problems in **CVX**PY. — Rodrigo de Azevedo, Oct 28 '19 at 22:26
@RodrigodeAzevedo Do you know a different library I might look at? Or maybe a way to make this problem convex? — George, Oct 28 '19 at 22:33
Mathematical questions should not be asked on SO. Try http://scicomp.stackexchange.com (which has a CVXPY tag). — Rodrigo de Azevedo, Oct 28 '19 at 22:37
Nonconvex QP are difficult and require a global solver (e.g. Baron, Couenne, Antigone, Cplex). There is also an interesting reformulation into a MIP problem (basically forming the KKT conditions). — Erwin Kalvelagen, Oct 30 '19 at 21:40
However we can exploit the variables x are binary. This yields a simpler MIP reformulation. — Erwin Kalvelagen, Oct 30 '19 at 22:30

Erwin Kalvelagen · Accepted Answer · 2019-10-31T18:45:34.627

The problem with your model is that max x'Qx is non-convex. As we have binary variables x we can use a trick.

Define

y(i,j) = x(i)*x(j)

as extra binary variable. Then we can write

sum((i,j), x(i)*Q(i,j)*x(j))

as

sum((i,j), y(i,j)*Q(i,j))

The binary multiplication y(i,j) = x(i)*x(j) can be linearized as:

 y(i,j) <= x(i)
 y(i,j) <= x(j)
 y(i,j) >= x(i)+x(j)-1

With this reformulation we have a completely linear model. It is a MIP as we have binary variables.

We can do this in CVXPY as:

import numpy as np
import cvxpy as cp

# Generate a random non-trivial quadratic program.

n = 10 # number of options

np.random.seed(1)
mu = np.random.randn(n) # expected means
var_covar = np.random.randn(n,n) # variance-covariance matrix
var_covar = var_covar.T.dot(var_covar) # cont'd
bench_cov = np.random.randn(n) # n-length vector of cov(benchmark, returns)

lamd = 0.01 # risk tolerance

e = np.ones((1,n))

x = cp.Variable((n,1), "x", boolean=True)
y = cp.Variable((n,n), "y", boolean=True)


prob = cp.Problem(cp.Maximize(mu.T@x + lamd * (cp.sum(cp.multiply(y,var_covar)) -2*bench_cov.T@x) ),
                  [y <= x@e, y <= (x@e).T, y >= x@e + (x@e).T - e.T@e, cp.sum(x)==4 ])

prob.solve(solver=cp.ECOS_BB)
print("status",prob.status)
print("obj",prob.value)
print("x",x.value)

This gives the result:

status optimal
obj 4.765120794509871
x [[1.00000000e+00]
 [3.52931931e-10]
 [3.80644178e-10]
 [2.53300872e-10]
 [9.99999999e-01]
 [1.79871537e-10]
 [1.00000000e+00]
 [3.46298454e-10]
 [9.99999999e-01]
 [1.00172269e-09]]

Notes:

You are encouraged to use a better MIP solver than ECOS_BB. For this model it gives the correct results, but it is somewhat of a toy solver and is known to give problems on more difficult data sets.
I don't understand the economics of the model. We are maximizing risk here. It may not be prudent to base your investment decisions on the results of this model.
Note that some high-end solvers (like Cplex and Gurobi) do this reformulation automatically. However CVXPY will not allow you to pass on the non-convex model to the solver.

I appreciate the detail of this response -- Yes, I've realized since I posted this that my quadratic term is non-convex. I ended up using one of the global non-convex solvers that you mentioned in your comment. That a really useful trick for turning QP problems into linear ones, thanks for making me aware of it :) -- Finally the economics of the model are a little more complicated.. I'm trying to create a portfolio to beat a benchmark. The E[benchmark] is above any E[portfolio], so instead of minimizing risk as you might do if the E[benchmark] was below an E[portfolio], I need to maximize it. — George, Oct 31 '19 at 16:59
I'd be happy to forward you the research I'm following if you're curious — George, Oct 31 '19 at 17:00
@ErwinKalvelagen Is this similar to the problem I'm seeing? https://stackoverflow.com/questions/68136486/cvxpy-error-when-trying-to-solve-a-convex-minimization-binary-programming-prob — geofflittle, Jun 26 '21 at 00:08

Why am I getting this DCPError?

1 Answers1