0

I've got this code of logistic regression but I can't seem to use gradient descent instead of the fmin_tnc function. The problem I'm trying to fix is that I want to create it from scratch and gradient descent seems like a viable way to do it. fmin_tnc is a straightforward way to get the optimal theta values but that isn't very intuitive. Here's the code:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import random

data = pd.read_csv("train.txt", header=None)
testing_data = pd.read_csv("test.txt")

# X = feature values, all the columns except the last column
X = data[data.columns[:-1]]

# y = target values, last column of the data frame
y = data.iloc[:, -1]

X = np.c_[np.ones((X.shape[0], 1)), X]
y = y[:, np.newaxis]
theta = np.zeros((X.shape[1], 1))

def sigmoid(x):
    return (1 / (1 + np.exp(-x)))

def net_input(theta, x):
    return np.dot(x, theta)

def probability(theta, x):
    return sigmoid(net_input(theta, x))

def cost_function(theta, x, y):
    m = x.shape[0]
    total_cost = -(1 / m) * np.sum(
        y * np.log(probability(theta, x)) + (1 - y) * np.log(
            1 - probability(theta, x)))
    return total_cost

def gradient(theta, x, y):
    # Computes the gradient of the cost function at the point theta
    m = x.shape[0]
    return (1 / m) * np.dot(x.T, sigmoid(net_input(theta,   x)) - y)

def fit(x, y, theta):
    opt_weights = fmin_tnc(func=cost_function, x0=theta,
                  fprime=gradient,args=(x, y.flatten()))
    return opt_weights[0]

parameters = fit(X, y, theta)

def predict(x):
    theta = parameters[:, np.newaxis]
    return probability(theta, x)

def accuracy(x, actual_classes, probab_threshold=0.5):
    predicted_classes = (predict(x) >= 
                         probab_threshold).astype(int)
    predicted_classes = predicted_classes.flatten()
    accuracy = np.mean(predicted_classes == actual_classes)
    return accuracy * 100

accuracy(X, y.flatten())
Flavia Giammarino
  • 7,987
  • 11
  • 30
  • 40
Sankhojyoti Halder
  • 13
  • 1
  • 3
  • Note that your code doesn't seem to import `fmin_tnc` from `scipy` – SomethingSomething Nov 10 '21 at 10:24
  • Are you willing to implement Gradient Decent by yourself? Why not use an off-the-shelf optimizer? [Take a look at all the minimization methods in scipy](https://docs.scipy.org/doc/scipy/reference/optimize.html#local-multivariate-optimization). Didn't see there GD directly, but many others that may share the same concept. – SomethingSomething Nov 10 '21 at 10:31
  • There is a more detailed documentation for the scipy optimization methods, including the math [here](https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html) – SomethingSomething Nov 10 '21 at 10:34

0 Answers0