Portfolio Optimization with CVXPY constraint has DCP error

Question

I am optimizing a portfolio where I maximize active return. I also have four constraints: the sum of weights must be equal to 1, all weights must be less than or equal to 0.05, all weights must be greater than or equal to 0.0005, and active risk must be equal less than or equal 7%.

I have 99 stocks. This means that I have 3 matrices being used in my calculations. alphas (expected returns) is a matrix with a shape of (99,1). W_bench is the weight each stock has in the benchmark and it has the same shape as alphas. V is a covariance matrix with a shape of (99,99). As can be seen in my code below, Active_Risk is what some people call tracking error. The code below is how I am setting up the optimization:

weights = cp.Variable((99,1))
Active_Return = weights.T @ alphas
Active_Risk = cp.sqrt((weights - W_bench).T @ V @ (weights - W_bench))
constraints = [cp.sum(weights) == 1, 0.07 >= Active_Risk, 0.0005 <= weights, weights <= 0.05]
prob = cp.Problem(cp.Maximize(Active_Return), constraints)

I can successfully solve this problem using excel's solver, though it takes quite a long time. However, I can't seem to get it to work using CVXPY on Python. I would happily upload my data so that others can help me fix the problem but I do not know how to do that. Also I considered making a smaller version of this problem with random data but I am afraid an optimal solution might not be feasible.

When I run my code result = prob.solve() I get the following error:

---------------------------------------------------------------------------
DCPError                                  Traceback (most recent call last)
<ipython-input-15-e01c2b878783> in <module>
      1 # The optimal objective value is returned by `prob.solve()`.
----> 2 result = prob.solve()

/opt/anaconda3/lib/python3.7/site-packages/cvxpy/problems/problem.py in solve(self, *args, **kwargs)
    287         else:
    288             solve_func = Problem._solve
--> 289         return solve_func(self, *args, **kwargs)
    290 
    291     @classmethod

/opt/anaconda3/lib/python3.7/site-packages/cvxpy/problems/problem.py in _solve(self, solver, warm_start, verbose, parallel, gp, qcp, **kwargs)
    565                     solver, warm_start, verbose, **kwargs)
    566 
--> 567         self._construct_chains(solver=solver, gp=gp)
    568         data, solving_inverse_data = self._solving_chain.apply(
    569             self._intermediate_problem)

/opt/anaconda3/lib/python3.7/site-packages/cvxpy/problems/problem.py in _construct_chains(self, solver, gp)
    508 
    509             except Exception as e:
--> 510                 raise e
    511 
    512     def _solve(self,

/opt/anaconda3/lib/python3.7/site-packages/cvxpy/problems/problem.py in _construct_chains(self, solver, gp)
    497 
    498                 self._intermediate_chain = \
--> 499                     construct_intermediate_chain(self, candidate_solvers, gp=gp)
    500                 self._intermediate_problem, self._intermediate_inverse_data = \
    501                     self._intermediate_chain.apply(self)

/opt/anaconda3/lib/python3.7/site-packages/cvxpy/reductions/solvers/intermediate_chain.py in construct_intermediate_chain(problem, candidates, gp)
     68             append += ("\nHowever, the problem does follow DQCP rules. "
     69                        "Consider calling solve() with `qcp=True`.")
---> 70         raise DCPError("Problem does not follow DCP rules. Specifically:\n" + append)
     71 
     72     elif gp and not problem.is_dgp():

DCPError: Problem does not follow DCP rules. Specifically:
The following constraints are not DCP:
power(var58 + -[[5.39791975e-03]
 [1.26209297e-03]
 [8.00351893e-05]
 [5.26548876e-02]
 [7.59655721e-03]
 [2.57535232e-03]
 [5.03372951e-04]
 [1.47480336e-03]
 [1.38599909e-04]
 [5.74108704e-03]
 [2.17434475e-03]
 [2.60441630e-04]
 [2.15412708e-04]
 [7.40692609e-03]
 [1.87030133e-04]
 [1.99318918e-04]
 [4.49890561e-04]
 [5.54053889e-04]
 [2.40066410e-04]
 [3.25109445e-05]
 [6.29530293e-02]
 [9.26141788e-03]
 [9.06847951e-02]
 [3.82569606e-04]
 [3.28101977e-04]
 [9.46531949e-04]
 [3.65845535e-02]
 [6.64869333e-04]
 [1.82867338e-03]
 [1.16324940e-04]
 [9.39765398e-04]
 [3.12166211e-03]
 [5.94047522e-04]
 [2.93656014e-04]
 [8.15666860e-04]
 [1.53355187e-04]
 [5.43259663e-04]
 [2.11729826e-04]
 [1.25169822e-04]
 [7.45171899e-04]
 [3.65823985e-04]
 [5.55365318e-04]
 [7.91907415e-05]
 [5.30872851e-03]
 [8.73601572e-04]
 [9.36948807e-04]
 [1.03941556e-02]
 [2.95556711e-04]
 [7.67666485e-03]
 [2.14996147e-04]
 [3.91275909e-04]
 [2.27743262e-04]
 [2.31689803e-04]
 [6.84002188e-03]
 [7.09758365e-02]
 [2.27532699e-04]
 [8.05783401e-04]
 [4.63372193e-04]
 [1.91341960e-03]
 [3.45573641e-04]
 [2.30427458e-02]
 [8.18612558e-04]
 [1.43341614e-03]
 [1.53359342e-04]
 [1.72991720e-04]
 [3.30942207e-04]
 [2.12224011e-02]
 [2.93271371e-04]
 [5.22032722e-02]
 [2.96349926e-03]
 [6.83703630e-04]
 [3.92175651e-04]
 [1.55757896e-03]
 [2.73614114e-04]
 [7.23199807e-03]
 [1.06194086e-02]
 [1.89837834e-04]
 [3.06087369e-03]
 [2.11597860e-04]
 [2.87620087e-04]
 [3.61744658e-04]
 [4.09530072e-04]
 [3.72525152e-04]
 [2.96987842e-02]
 [3.82775868e-01]
 [1.09826720e-03]
 [1.49363231e-03]
 [2.34448326e-04]
 [6.51708448e-03]
 [3.43523667e-03]
 [2.72356446e-03]
 [1.01445242e-03]
 [3.21249033e-04]
 [1.73042944e-02]
 [2.56326349e-03]
 [1.53653802e-03]
 [2.31159852e-04]
 [1.11857284e-03]
 [1.00845275e-02]].T * [[ 1.88468032  0.08627036 -0.81308165 ... -2.0625901  -0.64064643
  -1.12708076]
 [ 0.08627036  0.33892061  0.07628398 ...  0.19302222  0.14115192
  -0.02078213]
 [-0.81308165  0.07628398  0.58311836 ...  1.07460175  0.38954524
   0.53023314]
 ...
 [-2.0625901   0.19302222  1.07460175 ...  2.86645017  0.95771264
   1.34641816]
 [-0.64064643  0.14115192  0.38954524 ...  0.95771264  0.47074258
   0.43406168]
 [-1.12708076 -0.02078213  0.53023314 ...  1.34641816  0.43406168
   0.7897458 ]] * (var58 + -[[5.39791975e-03]
 [1.26209297e-03]
 [8.00351893e-05]
 [5.26548876e-02]
 [7.59655721e-03]
 [2.57535232e-03]
 [5.03372951e-04]
 [1.47480336e-03]
 [1.38599909e-04]
 [5.74108704e-03]
 [2.17434475e-03]
 [2.60441630e-04]
 [2.15412708e-04]
 [7.40692609e-03]
 [1.87030133e-04]
 [1.99318918e-04]
 [4.49890561e-04]
 [5.54053889e-04]
 [2.40066410e-04]
 [3.25109445e-05]
 [6.29530293e-02]
 [9.26141788e-03]
 [9.06847951e-02]
 [3.82569606e-04]
 [3.28101977e-04]
 [9.46531949e-04]
 [3.65845535e-02]
 [6.64869333e-04]
 [1.82867338e-03]
 [1.16324940e-04]
 [9.39765398e-04]
 [3.12166211e-03]
 [5.94047522e-04]
 [2.93656014e-04]
 [8.15666860e-04]
 [1.53355187e-04]
 [5.43259663e-04]
 [2.11729826e-04]
 [1.25169822e-04]
 [7.45171899e-04]
 [3.65823985e-04]
 [5.55365318e-04]
 [7.91907415e-05]
 [5.30872851e-03]
 [8.73601572e-04]
 [9.36948807e-04]
 [1.03941556e-02]
 [2.95556711e-04]
 [7.67666485e-03]
 [2.14996147e-04]
 [3.91275909e-04]
 [2.27743262e-04]
 [2.31689803e-04]
 [6.84002188e-03]
 [7.09758365e-02]
 [2.27532699e-04]
 [8.05783401e-04]
 [4.63372193e-04]
 [1.91341960e-03]
 [3.45573641e-04]
 [2.30427458e-02]
 [8.18612558e-04]
 [1.43341614e-03]
 [1.53359342e-04]
 [1.72991720e-04]
 [3.30942207e-04]
 [2.12224011e-02]
 [2.93271371e-04]
 [5.22032722e-02]
 [2.96349926e-03]
 [6.83703630e-04]
 [3.92175651e-04]
 [1.55757896e-03]
 [2.73614114e-04]
 [7.23199807e-03]
 [1.06194086e-02]
 [1.89837834e-04]
 [3.06087369e-03]
 [2.11597860e-04]
 [2.87620087e-04]
 [3.61744658e-04]
 [4.09530072e-04]
 [3.72525152e-04]
 [2.96987842e-02]
 [3.82775868e-01]
 [1.09826720e-03]
 [1.49363231e-03]
 [2.34448326e-04]
 [6.51708448e-03]
 [3.43523667e-03]
 [2.72356446e-03]
 [1.01445242e-03]
 [3.21249033e-04]
 [1.73042944e-02]
 [2.56326349e-03]
 [1.53653802e-03]
 [2.31159852e-04]
 [1.11857284e-03]
 [1.00845275e-02]]), 1/2) <= 0.07 , because the following subexpressions are not:
|--  var58 + -[[5.39791975e-03]
 [1.26209297e-03]
 [8.00351893e-05]
 [5.26548876e-02]
 [7.59655721e-03]
 [2.57535232e-03]
 [5.03372951e-04]
 [1.47480336e-03]
 [1.38599909e-04]
 [5.74108704e-03]
 [2.17434475e-03]
 [2.60441630e-04]
 [2.15412708e-04]
 [7.40692609e-03]
 [1.87030133e-04]
 [1.99318918e-04]
 [4.49890561e-04]
 [5.54053889e-04]
 [2.40066410e-04]
 [3.25109445e-05]
 [6.29530293e-02]
 [9.26141788e-03]
 [9.06847951e-02]
 [3.82569606e-04]
 [3.28101977e-04]
 [9.46531949e-04]
 [3.65845535e-02]
 [6.64869333e-04]
 [1.82867338e-03]
 [1.16324940e-04]
 [9.39765398e-04]
 [3.12166211e-03]
 [5.94047522e-04]
 [2.93656014e-04]
 [8.15666860e-04]
 [1.53355187e-04]
 [5.43259663e-04]
 [2.11729826e-04]
 [1.25169822e-04]
 [7.45171899e-04]
 [3.65823985e-04]
 [5.55365318e-04]
 [7.91907415e-05]
 [5.30872851e-03]
 [8.73601572e-04]
 [9.36948807e-04]
 [1.03941556e-02]
 [2.95556711e-04]
 [7.67666485e-03]
 [2.14996147e-04]
 [3.91275909e-04]
 [2.27743262e-04]
 [2.31689803e-04]
 [6.84002188e-03]
 [7.09758365e-02]
 [2.27532699e-04]
 [8.05783401e-04]
 [4.63372193e-04]
 [1.91341960e-03]
 [3.45573641e-04]
 [2.30427458e-02]
 [8.18612558e-04]
 [1.43341614e-03]
 [1.53359342e-04]
 [1.72991720e-04]
 [3.30942207e-04]
 [2.12224011e-02]
 [2.93271371e-04]
 [5.22032722e-02]
 [2.96349926e-03]
 [6.83703630e-04]
 [3.92175651e-04]
 [1.55757896e-03]
 [2.73614114e-04]
 [7.23199807e-03]
 [1.06194086e-02]
 [1.89837834e-04]
 [3.06087369e-03]
 [2.11597860e-04]
 [2.87620087e-04]
 [3.61744658e-04]
 [4.09530072e-04]
 [3.72525152e-04]
 [2.96987842e-02]
 [3.82775868e-01]
 [1.09826720e-03]
 [1.49363231e-03]
 [2.34448326e-04]
 [6.51708448e-03]
 [3.43523667e-03]
 [2.72356446e-03]
 [1.01445242e-03]
 [3.21249033e-04]
 [1.73042944e-02]
 [2.56326349e-03]
 [1.53653802e-03]
 [2.31159852e-04]
 [1.11857284e-03]
 [1.00845275e-02]].T * [[ 1.88468032  0.08627036 -0.81308165 ... -2.0625901  -0.64064643
  -1.12708076]
 [ 0.08627036  0.33892061  0.07628398 ...  0.19302222  0.14115192
  -0.02078213]
 [-0.81308165  0.07628398  0.58311836 ...  1.07460175  0.38954524
   0.53023314]
 ...
 [-2.0625901   0.19302222  1.07460175 ...  2.86645017  0.95771264
   1.34641816]
 [-0.64064643  0.14115192  0.38954524 ...  0.95771264  0.47074258
   0.43406168]
 [-1.12708076 -0.02078213  0.53023314 ...  1.34641816  0.43406168
   0.7897458 ]] * (var58 + -[[5.39791975e-03]
 [1.26209297e-03]
 [8.00351893e-05]
 [5.26548876e-02]
 [7.59655721e-03]
 [2.57535232e-03]
 [5.03372951e-04]
 [1.47480336e-03]
 [1.38599909e-04]
 [5.74108704e-03]
 [2.17434475e-03]
 [2.60441630e-04]
 [2.15412708e-04]
 [7.40692609e-03]
 [1.87030133e-04]
 [1.99318918e-04]
 [4.49890561e-04]
 [5.54053889e-04]
 [2.40066410e-04]
 [3.25109445e-05]
 [6.29530293e-02]
 [9.26141788e-03]
 [9.06847951e-02]
 [3.82569606e-04]
 [3.28101977e-04]
 [9.46531949e-04]
 [3.65845535e-02]
 [6.64869333e-04]
 [1.82867338e-03]
 [1.16324940e-04]
 [9.39765398e-04]
 [3.12166211e-03]
 [5.94047522e-04]
 [2.93656014e-04]
 [8.15666860e-04]
 [1.53355187e-04]
 [5.43259663e-04]
 [2.11729826e-04]
 [1.25169822e-04]
 [7.45171899e-04]
 [3.65823985e-04]
 [5.55365318e-04]
 [7.91907415e-05]
 [5.30872851e-03]
 [8.73601572e-04]
 [9.36948807e-04]
 [1.03941556e-02]
 [2.95556711e-04]
 [7.67666485e-03]
 [2.14996147e-04]
 [3.91275909e-04]
 [2.27743262e-04]
 [2.31689803e-04]
 [6.84002188e-03]
 [7.09758365e-02]
 [2.27532699e-04]
 [8.05783401e-04]
 [4.63372193e-04]
 [1.91341960e-03]
 [3.45573641e-04]
 [2.30427458e-02]
 [8.18612558e-04]
 [1.43341614e-03]
 [1.53359342e-04]
 [1.72991720e-04]
 [3.30942207e-04]
 [2.12224011e-02]
 [2.93271371e-04]
 [5.22032722e-02]
 [2.96349926e-03]
 [6.83703630e-04]
 [3.92175651e-04]
 [1.55757896e-03]
 [2.73614114e-04]
 [7.23199807e-03]
 [1.06194086e-02]
 [1.89837834e-04]
 [3.06087369e-03]
 [2.11597860e-04]
 [2.87620087e-04]
 [3.61744658e-04]
 [4.09530072e-04]
 [3.72525152e-04]
 [2.96987842e-02]
 [3.82775868e-01]
 [1.09826720e-03]
 [1.49363231e-03]
 [2.34448326e-04]
 [6.51708448e-03]
 [3.43523667e-03]
 [2.72356446e-03]
 [1.01445242e-03]
 [3.21249033e-04]
 [1.73042944e-02]
 [2.56326349e-03]
 [1.53653802e-03]
 [2.31159852e-04]
 [1.11857284e-03]
 [1.00845275e-02]])

I am new to CVXPY and don't fully understand DCP rules. Any help would be appreciated. Thanks.

Answer 1

This question is in bad shape as everyone trying to make it run needs work you could have avoided by working on some self-contained example (and yes: even with your remark about some initial failed attempt, i think it should be easier to do that yourself than ask us for it).

There is also the factor context. DCP is a rule-based framework able to express lots of convex problems, but not all convex problems. There are some indications leading to the fact, that this problem is DCP-compatible, but a definite answer to that, again, will result in more work on our side.

This all leads to limited guarantees from my side about the following:

Your problem is to be found in:

Active_Risk = cp.sqrt((weights - W_bench).T @ V @ (weights - W_bench))

• this is a product of variables which is in general non-convex and therefore, handling the general case without further assumptions cannot be expressed in DCP
• you got additional assumptions like V being PSD (as it's a covariance-matrix), but cvxpy can't infer this given this form
• you will need to express this additional structure / assumptions to cvxpy more clearly

Instead of using:

cp.sqrt((weights - W_bench).T @ V @ (weights - W_bench))

you will need to use:

cp.quad_form((weights - W_bench), V)

I have a bad feeling about an outer cp.sqrt which is lost in above line, but you could try it.

It's important to see though, that using this non-sqrt quad_form can be corrected by changing your other model-components (scaling your parameters).

So instead of:

0.07 >= Active_Risk

you will have:

0.07^2 >= Active_Risk

You won't need to touch the objective (squared or not; the objective changes, but the solution vector will not).

Answer 2

I was able to get the optimization to work with the following code:

weights = cp.Variable((99,1))
Active_Return = weights.T @ alphas
Active_Risk = cp.quad_form((weights - W_bench), V)
constraints = [sum(weights) == 1, 0.07**2 >= Active_Risk, weights >= 0.0005, weights <= 0.05]
prob = cp.Problem(cp.Maximize(Active_Return), constraints)


result = prob.solve(qcp=True, verbose= False, solver= 'SCS', eps= 1e-10, max_iters = 100000, warm_start= True)

Note: I had several errors when either not specifying the solver or when using ECOS. SCS works perfectly but needs a large number of iterations to find the optimal solution. Either way it only takes a few seconds for the code to run.