sklearn: SVR fails to generalise adder function

Question

This is my SVR to learn adder function (y=x1 + x2):

%reset -f

#Libs
from sklearn import svm;

#PROGRAMME ENTRY POINT==========================================================
#Data, addition
#Exp[I] = sum(Inp[I])
Inp = [[1,2],[3,4],[5,6],[7,8],[9,0]];
Exp = [ 3,    7,    11,   15,   9   ];

#Train
Model = svm.SVR(kernel="poly",degree=3);
Model.fit(Inp,Exp);

#Infer
print("Input values are those in the train data:");
print(f"1 + 2 = {Model.predict([[1,2]])[0]:.6f}");

print("\nInput values are those in the train data:");
print(f"5 + 6 = {Model.predict([[5,6]])[0]:.6f}");

print("\nInput values are those NOT in the train data, but in range:");
print(f"5 + 5 = {Model.predict([[5,5]])[0]:.6f}");

print("\nInput values are those NOT in the train data, and OUT of range:");
print(f"9 + 1 = {Model.predict([[9,1]])[0]:.6f}");
#EOF

But the result isn't what expected for:

Input values are those in the train data:
1 + 2 = 6.007171

Input values are those in the train data:
5 + 6 = 9.595818

Input values are those NOT in the train data, but in range:
5 + 5 = 8.533934

Input values are those NOT in the train data, and OUT of range:
9 + 1 = 9.170507

Is it possible for sklearn SVR to generalise the adder function? What should be changed in the code above to make SVR learn x1+x2?

Matus Dubrava · Accepted Answer · 2020-07-18T06:07:53.613

Polynomial kernel of third degree simply have too much variance to correctly predict such a simple function, especially on such a small dataset. This is based on bias/variance tradeoff. Your model is loosing on variance while getting almost nothing on bias in this case (your function is overly complex). The same goes even for lower order polynomials and radial basis function.

Lowering the variance of the model will do the trick. Just use linear kernel.

Model = svm.SVR(kernel="linear")

Results for SVM with linear kernel are:

Input values are those in the train data:
1 + 2 = 3.100000

Input values are those in the train data:
5 + 6 = 10.966667

Input values are those NOT in the train data, but in range:
5 + 5 = 9.983333

Input values are those NOT in the train data, and OUT of range:
9 + 1 = 9.983333

sklearn: SVR fails to generalise adder function

1 Answers1