2

I'm beginner in optimization and welcome any guide in this field.

I have 15 matrices (i.e., Di of size (n*m)) and want to find best weights (i.e., wi) for weighted averaging them and make a better matrix that is more similar to one given matrix (i.e., Dt).

In fact my objective function is like it:

min [norm2(sum(wi * Di) - Dt) + norm2(W)]
for i=1 ... 15    s.t. sum(wi) = 1 , wi >= 0

How can I optimize this function in Matlab?

Shai
  • 111,146
  • 38
  • 238
  • 371
roya
  • 23
  • 4

1 Answers1

2

You are describing a simple Quadratic programming, that can be easily optimized using Matlab's quadprog.

Here how it goes:

You objective function is [norm2(sum(wi * Di) - Dt) + norm2(W)] subject to some linear constraints on w. Let's re-write it using some simplified notations. Let w be a 15-by-1 vector of unknowns. Let D be an n*m-by-15 matrix (each column is one of the Di matrices you have - written as a single column), and Dt is a n*m-by-1 vector (same as your Dt but written as a column vector). Now some linear algebra (using the fact that ||x||^2 = x'*x and that argmin x is equivalent to argmin x^2)

[norm2(sum(wi * Di) - Dt)^2 + norm2(W)^2] =
(D*w-Dt)'*(D*w-Dt) + w'*w =
w'D'Dw - 2w'D'Dt + Dt'Dt + w'w =
w'(D'D+I)w - 2w'D'Dt + Dt'Dt

The last term Dt'Dt is constant w.r.t w and therefore can be discarded during minimization, leaving you with

H = 2*(D'*D+eye(15));
f = -2*Dt'*D;

As for the constraint sum(w)=1, this can easily be defined by

Aeq = ones(1,15);
beq = 1;

And a lower bound lb = zeros(15,1) will ensure that all w_i>=0.

And the quadratic optimization:

w = quadprog( H, f, [], [], Aeq, beq, lb );

Should do the trick for you!

Shai
  • 111,146
  • 38
  • 238
  • 371
  • @NereaGonzalezVazquez in this particular case quadratic programming is more suited than genetic algorithm. – Shai Apr 13 '15 at 09:00
  • Thanks a lot! Your explanantion was very useful for me. – roya Apr 14 '15 at 07:43
  • But I have a problem with H. For calculating term D' * D, I have two 3d matrix (i.e., (m * n * 15) * (n * m * 15)). I did D(:,:,i)' * D(:,:,i) for i:1...15 in a loop, and the result was a m * m * 15 matrix. Is it true? If yes, how can I add it with eye(15)? – roya Apr 14 '15 at 07:54
  • @roya - please read my solution carefully: I convert the `m`-by-`n` matrices to `m*n`-by-1 vectors (simply reshaping them) this will simplify the algebra and allows for simple solution. – Shai Apr 14 '15 at 08:44
  • Oh. you are right! I hadn't understood it. Thanks :) – roya Apr 14 '15 at 09:01
  • Finally I finished it but the result isn't meaningful and one of wi is 1 randomly and the others are zeros. In my problem the dimension of each Di is 4500*4500. Do you think it is too big and optimization is failed because of dimensionality? – roya Apr 14 '15 at 09:46
  • @roya (1) the dimensionality of `Di` is meaningless - the effective dimension of your problem is 15. As you can see the coefficients of `quadprog` are of size 15-by-15 and 15-by-1. (2) Try supplying `quadprog` with `x0` - an initial guess `x0=ones(1,15)/15` - does this affect the result? (3) what are the typical values of `H` and `f`? you might have a problem with coefficients that are very large causing numerical instabilities. – Shai Apr 14 '15 at 10:21
  • @roya: coming to think about it, you have `L2` regularization term on `w` **and** `L1` constraint (`sum(w_i)=1`) - in this case it is not surprising you get `w` that is binary. Try to remove the `L2` regularization term (the `eye(15)` component of `H`) and see what happens. – Shai Apr 14 '15 at 10:24
  • removing L2 regularization term made no difference, but removing L1 constraint let wi select difference values between 0 to 1.5. Now I must do some other works to can evaluate these values are good or not. Thank you for your useful notes. :) – roya Apr 15 '15 at 06:37
  • Sorry for my more question. I used from learned W in my work. But it seems the learned weights aren't suitable. The iteration parameter of quadprog's output is only 4. Is it usual? – roya Apr 21 '15 at 07:20
  • @roya I do not understand your question. why your learned weights are not okay? – Shai Apr 21 '15 at 07:23
  • I mean these Wi which are learned from training set didn't work for my test set. I.e., sum(wi * D'i) - D't is bigger than sum(D'i) - D't. where D'i and D't are extracted from test set. – roya Apr 21 '15 at 07:58
  • @roya machine learning can be tricky sometimes... is it possible that the training set and test set are not sampled from the same distribution? – Shai Apr 21 '15 at 08:04
  • My dataset is Corel 5K that is the most famous in image annotation and its training and testing set are fixed. But now I tested Wi in training set and it was good for it. Yes, it seems learning weights fails because of my dataset. Thank you anyway :) – roya Apr 21 '15 at 08:14