How to minimize this function and apply it with backpropagation algorithm

Question

I'm curently working with backpropagation algorithm For mining data. I'm trying to apply several algorithm from related paper to my cases. Here, I find that, one of them said that "Backpropagation algorithm is applied to update the weights(w, v) and minimizing the following function":

θ(w,v) = F(w,v) + P(w,v)

where F(w,v) is cross entropy function and P(w,v) is penalty term. I've not got the meaning of this, since I've found that "minimizing" in programming term has several meaning.

Please give me some suggestion, what should I do with the function, and how to applied it into backpropagation.

Thank you.

Answer 1

Backprop is an algorithm for computing derivatives/gradients of a loss function wrt. a neural network's weights. When combined with an optimization algorithm (commonly gradient descent or conjugate gradient, although other algorithms are used as well), it can be used to find the NN weights that minimize the loss function on a training set.

I.e., the suggestion is that you train a neural net by minimizing regularized cross-entropy loss. That's what's usually meant by "training" a neural net for classification and what many NN libraries/toolkits actually do.

(Of course, if P is some non-standard penalty term, you may have to implement backprop yourself or find a toolkit that is extensible; non-differentiable penalty terms may also require changes to the optimization algorithm.)