Minimizing SSE using Scipy.optimize minimize

Question

I am trying to optimize SSE (sum of squared error) of a function using scipy.optimize. To test with, I created a simple problem as below code. But the optimized parameters output by scipy never makes SSE=0. Can someone help me to understand, where am I going wrong.

I tried to cross check with the SSE calculated by my code with the one computed in excel. It matched. Then I used minimize function to minimize that SSE function, the ones computed by Scipy is not matching with the hand calculated ones. The function I used to is of form (y=ax+b). Below is the code

import numpy as np
from scipy.optimize import minimize


e=np.array([0,2])
sig1=np.array([0,200])
k = [10,10]
#n = 0.2
coe=np.array([k[0],k[1]])

def sig2(e):
    v=(k[0]*e)+ k[1]
    SEzip = zip(sig1, v)
    sse = 0
    for y in SEzip:
        sse += np.power((y[0] - y[1]),2)
    return sse

print (sig2(e))
def f(coe):
   print(coe)
   return f
result = minimize(sig2,coe,method='Nelder-Mead',callback=(f),options={'xtol': 1e-6,'ftol':1e-06,'maxiter':50000,'disp': True,'adaptive' : True})

print(result)

It will be of great help if someone can help me out to fix this issue. — Sajna, Apr 28 '19 at 16:37

score 0 · Accepted Answer · answered Apr 28 '19 at 10:35

You were printing your x0 aka coe in here, I edited your code and shortened your objective function sig2() into one line then edited your callback to display the variable tested and its equivalent objective function value. Now you can clearly see that sse=0 is reached.

import numpy as np
from scipy.optimize import minimize

# for prettier numpy prints
np.set_printoptions(precision = 6)

# init 
e    = np.array([0,2])
sig1 = np.array([0,200])
k    = [10, 10]
coe  = np.array([k[0], k[1]])

# define objective function
def sig2(e):
    return sum([np.power((y[0] - y[1]), 2) for y in zip(sig1, (k[0]*e)+ k[1])])

# define callback
def f(e):
   print("e: %25s | sig2(e): %5s" % (e,round(sig2(e), 6)))

# optimize
result = minimize(sig2,
                  coe,
                  method   = 'Nelder-Mead',
                  callback = f,
                  options  = {'xtol': 1e-6,'ftol':1e-06,
                              'maxiter':50000,'disp': True,'adaptive' : True})

print(result)

Output:

...
e:     [-1.000053 18.999751] | sig2(e): 6e-06
e:     [-1.000062 19.000109] | sig2(e): 2e-06
e:     [-1.000062 19.000109] | sig2(e): 2e-06
e:     [-1.000062 19.000109] | sig2(e): 2e-06
e:     [-0.999934 18.999981] | sig2(e):   0.0
e:     [-1.000049 18.999979] | sig2(e):   0.0
e:     [-1.000027 19.000044] | sig2(e):   0.0
e:     [-0.999986 18.999996] | sig2(e):   0.0
e:     [-0.999986 18.999996] | sig2(e):   0.0
e:     [-0.999986 18.999996] | sig2(e):   0.0
e:     [-1.000009 18.999993] | sig2(e):   0.0
e:     [-1.000009 18.999993] | sig2(e):   0.0
e:     [-0.999995 19.      ] | sig2(e):   0.0
e:     [-0.999995 19.      ] | sig2(e):   0.0
e:     [-1.000003 18.999998] | sig2(e):   0.0
e:     [-1.       19.000002] | sig2(e):   0.0
e:     [-0.999998 19.      ] | sig2(e):   0.0
e:     [-1.000001 18.999999] | sig2(e):   0.0
e:     [-1.       19.000001] | sig2(e):   0.0
e:     [-0.999999 19.      ] | sig2(e):   0.0
e:                 [-1. 19.] | sig2(e):   0.0
e:                 [-1. 19.] | sig2(e):   0.0
e:                 [-1. 19.] | sig2(e):   0.0
Optimization terminated successfully.
         Current function value: 0.000000
         Iterations: 56
         Function evaluations: 110
 final_simplex: (array([[-1., 19.],
       [-1., 19.],
       [-1., 19.]]), array([6.221143e-12, 1.914559e-11, 1.946860e-11]))
           fun: 6.2211434216849394e-12
       message: 'Optimization terminated successfully.'
          nfev: 110
           nit: 56
        status: 0
       success: True
             x: array([-1., 19.])

Thanks for your response. But I see the predicted optimum is not correct. Though Scipy output shows sse=0, if I substitute the predicted values, sse=33850.. The optimal value should be (100,0). — Sajna, Apr 28 '19 at 11:42
Where are you substituting the values? `sig2(np.array([-1, 19])) = 0` and `sig2(np.array([100, 0])) = 1056200 `? — SuperKogito, Apr 28 '19 at 11:51
Yes, I am manually substituting on the same equation to cross check the results. I checked in excel sheet as well — Sajna, Apr 28 '19 at 12:03
In that case your implementation of `sig2()` does not match your function that you are substituting into. Good luck. — SuperKogito, Apr 28 '19 at 12:14
when i print `sig2()` the first value is perfectly matching with the hand calculation. This means that the function is correct. — Sajna, Apr 28 '19 at 12:37
Just because one value somehow matches, it does not mean it is correct. You should check at least for 3 random values. — SuperKogito, Apr 28 '19 at 12:39
I tried with many random pairs for `k[0]` & `k[1]` and `print(sig2(e)`. all the values are computed correctly. Is there something missing on `scipy.optimize.minimze` function? — Sajna, Apr 28 '19 at 12:51
Thanks a lot for your support. I was passing wrong variable into the function.. sig2(e) should be sig2(k). Now it is working perfect!!! — Sajna, Apr 29 '19 at 12:41

Minimizing SSE using Scipy.optimize minimize

1 Answers1