What is the easiest to implement linear regression algorithm?

Question

I want to implement single variable regression using ordinary least squares. I have no access to linear algebra or calculus libraries, so any matrix operations or differentiation methods needs to be implemented by me. What is the least complex method?

Answer 1

John D. Cook has an excelent post on the subject with a simple C++ implementation. His implementation uses constant memory and can be parallelized with little effort.

I wrote a simple Python version of it. Use with caution, there may be bugs:

class Regression:
    def __init__(self):
        self.n = 0.0
        self.sXY = 0.0
        self.xM1 = 0.0
        self.xM2 = 0.0
        self.yM1 = 0.0
        self.yM2 = 0.0

    def add(self, x, y):
        self.sXY += (self.xM1 - x) * (self.yM1 - y) * self.n / (self.n + 1.0);

        n1 = self.n;
        self.n+=1;

        xdelta = x - self.xM1;
        xdelta_n = xdelta / self.n;
        self.xM1 += xdelta_n;
        self.xM2 += xdelta * xdelta_n * n1;

        ydelta = y - self.yM1;
        ydelta_n = ydelta / self.n;
        self.yM1 += ydelta_n;
        self.yM2 += ydelta * ydelta_n * n1;

    def count(self):
        return self.n

    def slope(self):
        return self.sXY / self.xM2

    def intercept(self):
        return self.yM1 - (self.sXY / self.xM2) * self.xM1

    def correlation(self):
        return self.sXY / (self.xM2**0.5 * self.yM2**0.5)

    def covariance(self):
        return self.sXY / self.n

r = Regression()

r.add(1, 2)
r.add(4, 9)
r.add(16, 17)
r.add(17, 13)
r.add(21, 11)

print 'Count:', r.count()
print 'Slope:', r.slope()
print 'Intercept:', r.intercept()
print 'Correlation:', r.correlation()
print 'Covariance:', r.covariance()