Quadratic Programming Setting up Constraint

Question

I am trying to set up a simple OLS model with constraints on the coefficients in R. The code below is working. However, this demonstrates

y = c + a1x1 + a2x2 + a3x3 with constraint a1+a2 = 1

I would like to revise this constraint to: a1*a2 - a3 = 0

thanks for your help!

WORKING CODE:

'''

    set.seed(1000) 
    n <- 20
    x1 <- seq(100,length.out=n)+rnorm(n,0,2)
    x2 <- seq(50,length.out=n)+rnorm(n,0,2)
    x3 <- seq(10,length.out=n)+rnorm(n,0,2)
    constant <- 100
    ymat <- constant + .5*x1 + .5*x2 + .75*x3 + rnorm(n,0,4)
    xmat <- cbind(x1,x2,x3)

    X <- cbind(rep(1,n),xmat) # explicitly include vector for constant
    bh <- solve(t(X)%*%X)%*%t(X)%*%ymat

    XX <- solve(t(X)%*%X)
    cmat <- matrix(1,1,1) 
    Q <- matrix(c(0,1,1,0),ncol(X),1) # a1+a2=1 for y = c + a1x1 + a2x2 + a3x3
    bc <- bh-XX%*%Q%*%solve(t(Q)%*%XX%*%Q)%*%(t(Q)%*%bh-cmat)

   library(quadprog)
   d <- t(ymat) %*% X
   Rinv = solve(chol(t(X)%*%X)) 
   qp <- solve.QP(Dmat=Rinv, dvec=d, Amat=Q, bvec=cmat, meq=1, factorized=TRUE)
   qp

   cbind(bh,qp$unconstrained.solution)
   cbind(bc,qp$solution)

'''

G. Grothendieck · Answer 1 · 2019-11-22T22:31:24.253

Assuming the problem is to minimize || ymat - X b || ^2 subject to b[2] * b[3] == b[4] we can substitute for b[4] giving the unconstrained nls problem shown below. b below is the first 3 elements of b and we can get b[4] by multiplying the last two elements of b below together. No packages are used.

fm <- nls(ymat ~ X %*% c(b, b[2] * b[3]), start = list(b = 0:2))
fm

giving:

Nonlinear regression model
  model: ymat ~ X %*% c(b, b[2] * b[3])
   data: parent.frame()
     b1      b2      b3 
76.9718  0.6275  0.7598 
 residual sum-of-squares: 204

Number of iterations to convergence: 4 
Achieved convergence tolerance: 6.555e-06

To compute b4

prod(coef(fm)[-1])
## [1] 0.476805

Note

In a similar way the original problem (to minimize the same objective but with the original constraint) can be reduced to an unconstrained problem and solved using nls via substitution:

nls(ymat ~ X %*% c(b[1], b[2], 1-b[2], b[3]), start = list(b = 0:2))

giving:

Nonlinear regression model
  model: ymat ~ X %*% c(b[1], b[2], 1 - b[2], b[3])
   data: parent.frame()
      b1       b2       b3 
105.3186   0.3931   0.7964 
 residual sum-of-squares: 222.3

Number of iterations to convergence: 1 
Achieved convergence tolerance: 4.838e-08

It would even be possible to reparameterize to make this original problem solvable by lm

lm(ymat ~ I(X[, 2] - X[, 3]) + X[, 4] + offset(X[, 3]))

giving

Call:
  lm(formula = ymat ~ I(X[, 2] - X[, 3]) + X[, 4] + offset(X[, 3]))

Coefficients:
       (Intercept)  I(X[, 2] - X[, 3])              X[, 4]  
          105.3186              0.3931              0.7964

score 0 · Answer 2 · answered Jan 10 '20 at 14:32

G. grothendieck - thank you for your response. Unfortunately this didn't work for me.

I decided to work out the Lagrangian long handed, which turned out too complicated for me to solve.

Then realized,

a1*a2-a3 =0 
a1*a2 = a3
ln(a1*a2)= ln(a3)
ln(a1) + ln(a2) -ln(a3) = 0

This leaves me with an additive constraint which I can solve with the quadprog package.

score 0 · Answer 3 · answered Jan 10 '20 at 15:28

Maybe you can try the code below, using fmincon()

library(pracma)
library(NlcOptim)
# define objective function
fn <- function(v) norm(ymat- as.vector( xmat %*% v),"2")
# the constraint a1*a2 - a3 = 0
heq1 = function(v) prod(v[1:2])-v[3] 
# solve a1, a2 and a3 
res <- fmincon(0:2,fn,heq = heq1)

such that

> res$par
[1]  1.9043754 -0.1781830 -0.3393272

Quadratic Programming Setting up Constraint

3 Answers3

Note