How to vectorize equations?

Question

I'm trying to implement the Softmax regression algorithm to solve the K-classifier problem after watching Professor Andrew Ng's lectures on GLM. I thought I understood everything he was saying until it finally came to writing the code to implement the cost function for Softmax regression, which is as follows:

The problem I am having is trying to figure out a way to vectorize this. Again I thought I understood how to go about vectorizing equations like this since I was able to do it for linear and logistic regression, but after looking at that formula I am stuck.

While I would love to figure out a vectorized solution for this (I realize there is a similar question posted already: Vectorized Implementation of Softmax Regression), what I am more interested in is whether any of you can tell me a way (your way) to methodically convert equations like this into vectorized forms. For example, for those of you who are experts or seasoned veterans in ML, when you read of new algorithms in the literature for the first time, and see them written in similar notation to the equation above, how do you go about converting them to vectorized forms?

I realize I might be coming off as being like the student who is asking Mozart, "How do you play the piano so well?" But my question is simply motivated from a desire to become better at this material, and assuming that not everyone was born knowing how to vectorize equations, and so someone out there must have devised their own system, and if so, please share! Many thanks in advance!

Cheers

Answer 1

The help files that come with Octave have this entry:

19.1 Basic Vectorization

To a very good first approximation, the goal in vectorization is to write code that avoids loops and uses whole-array operations. As a trivial example, consider

 for i = 1:n
   for j = 1:m
     c(i,j) = a(i,j) + b(i,j);
   endfor
 endfor

compared to the much simpler

 c = a + b;

This isn't merely easier to write; it is also internally much easier to optimize. Octave delegates this operation to an underlying implementation which, among other optimizations, may use special vector hardware instructions or could conceivably even perform the additions in parallel. In general, if the code is vectorized, the underlying implementation has more freedom about the assumptions it can make in order to achieve faster execution.

This is especially important for loops with "cheap" bodies. Often it suffices to vectorize just the innermost loop to get acceptable performance. A general rule of thumb is that the "order" of the vectorized body should be greater or equal to the "order" of the enclosing loop.

As a less trivial example, instead of

 for i = 1:n-1
   a(i) = b(i+1) - b(i);
 endfor

write

 a = b(2:n) - b(1:n-1);

This shows an important general concept about using arrays for indexing instead of looping over an index variable.  Index Expressions. Also use boolean indexing generously. If a condition needs to be tested, this condition can also be written as a boolean index. For instance, instead of

 for i = 1:n
   if (a(i) > 5)
     a(i) -= 20
   endif
 endfor

write

 a(a>5) -= 20;

which exploits the fact that 'a > 5' produces a boolean index.

Use elementwise vector operators whenever possible to avoid looping (operators like '.*' and '.^').  Arithmetic Ops. For simple inline functions, the 'vectorize' function can do this automatically.

-- Built-in Function: vectorize (FUN) Create a vectorized version of the inline function FUN by replacing all occurrences of '', '/', etc., with '.', './', etc.

 This may be useful, for example, when using inline functions with
 numerical integration or optimization where a vector-valued
 function is expected.

      fcn = vectorize (inline ("x^2 - 1"))
         => fcn = f(x) = x.^2 - 1
      quadv (fcn, 0, 3)
         => 6

 See also:  inline,  formula,
  argnames.

Also exploit broadcasting in these elementwise operators both to avoid looping and unnecessary intermediate memory allocations.
 Broadcasting.

Use built-in and library functions if possible. Built-in and compiled functions are very fast. Even with an m-file library function, chances are good that it is already optimized, or will be optimized more in a future release.

For instance, even better than

 a = b(2:n) - b(1:n-1);

is

 a = diff (b);

Most Octave functions are written with vector and array arguments in mind. If you find yourself writing a loop with a very simple operation, chances are that such a function already exists. The following functions occur frequently in vectorized code:

• Index manipulation

  * find
  
  * sub2ind
  
  * ind2sub
  
  * sort
  
  * unique
  
  * lookup
  
  * ifelse / merge
  

• Repetition

  * repmat
  
  * repelems
  

• Vectorized arithmetic

  * sum
  
  * prod
  
  * cumsum
  
  * cumprod
  
  * sumsq
  
  * diff
  
  * dot
  
  * cummax
  
  * cummin
  

• Shape of higher dimensional arrays

  * reshape
  
  * resize
  
  * permute
  
  * squeeze
  
  * deal
  

---

Also look at these pages from a Stanford ML wiki for some more guidance with examples.

http://ufldl.stanford.edu/wiki/index.php/Vectorization

http://ufldl.stanford.edu/wiki/index.php/Logistic_Regression_Vectorization_Example

http://ufldl.stanford.edu/wiki/index.php/Neural_Network_Vectorization

Answer 2

This one looks pretty hard to vectorize since you are doing exponentials inside of your summations. I assume you are raising e to arbitrary powers. What you can vectorize is the second term of the expression \sum \sum theta ^2 just make sure to use .* operator in matlab enter link description here to computer \theta ^2

Same goes for the inner terms of the ratio of the that goes into the logarithm. \theta ' x^(i) is vectorizable expression.

You might also benefit from a memoization or dynamic programming technique and try to reuse the results of computations of e^\theta' x^(i).

Generally in my experience the way to vectorize is first to get non-vectorized implementation working. Then try to vectorize the most obvious parts of your computation. At every step tweak your function very little and always check if you get the same result as non-vectorized computation. Also, having multiple test cases is very helpful.