Loss of logistic regression model not decreasing through gradient descent

Question

I'm a beginner to machine learning and have been trying to implement gradient descent to try and optimize the weights of my model. I am trying to develop the model from scratch and I have reviewed a lot of code online but my implementation still doesnt seem to decrease the loss of the model with the loss oscillating between 0.2 and 0.1. The loss function I used is L = (y - hypothesis)**2. Any help would be appreciated

    for z in range(self.iterations):
        print(z)
        cost = 0
        for x in range(self.batch_size):
            derivatives = np.zeros(self.num_weights)
            ran = self.random_row()
            row = self.X.iloc[[ran]]
            cost += self.loss(row, self.y[ran])
            error = self.y[ran] - self.predict(row)
            for i in range(len(derivatives)):
                derivatives[i] = derivatives[i] + (error * (row.iloc[0,i] * -2))
                derivatives[i] = derivatives[i] * learning_rate
                self.weights[i] = self.weights[i] - derivatives[i]

How does the loss behave? Oscillating or diverging? What is the learning rate? — Frightera, Feb 07 '21 at 11:22
Lowering the LR usually solves the oscillation, however I suggest you to add some details to your question. — Frightera, Feb 07 '21 at 11:48

score 1 · Answer 1 · edited May 27 '21 at 16:14

This is the incorrect loss function. For binary/two-class logistic regression you should use the cost function of

binary logistic regression cost equation

where h is the hypothesis.

You can find an intuition for the cost function and an explaination of why it is what it is in the 'Cost function intuition' section of this article here.

With this cost function, you will need to use a different gradient: gradient equation .

Note that here different notation is used and the cost function is represented by J(θ) where θ is a vector of weights for this model. A derivation of this equation can be found here.

In general, here are some helpful articles about implementing logistic regression:

score 0 · Answer 2 · answered Feb 07 '21 at 12:05

0

iam beginner in learning machine learning. I usually use sklearn library in python. you can put the code at head line from sklearn.linear model import SGDlassifier then the SGDClassifer fit with your data by using SGDClassifier.fit(X_train,y_train)

answered Feb 07 '21 at 12:05

Handri Mauludin Maulana

75
1
14

Loss of logistic regression model not decreasing through gradient descent

2 Answers2