19

I'm trying to convert some old code from using sklearn to Keras implementation. Since it is crucial to maintain the same way of operation, I want to understand if I'm doing it correctly.

I've converted most of the code already, however I'm having trouble with sklearn.svm SVC classifier conversion. Here is how it looks right now:

from sklearn.svm import SVC
model = SVC(kernel='linear', probability=True)
model.fit(X, Y_labels)

Super easy, right. However, I couldn't find the analog of SVC classifier in Keras. So, what I've tried is this:

from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(64, activation='relu'))
model.add(Dense(1, activation='softmax'))
model.compile(loss='squared_hinge',
              optimizer='adadelta',
              metrics=['accuracy'])
model.fit(X, Y_labels)

But, I think that it is not correct by any means. Could you, please, help me find an alternative of the SVC classifier from sklearn in Keras?

Thank you.

desertnaut
  • 57,590
  • 26
  • 140
  • 166
none32
  • 515
  • 1
  • 5
  • 21
  • What exactly do you mean by "alternative"? Keras is specifically a neural network framework, and it does not include SVM functionality... – desertnaut Jan 29 '19 at 09:49
  • 1
    Yes, it does not come out of the box, but you can compose a model that will be the alternative of SVM loss function found in sklearn-kit. This kind of the model is proposed below in the answer. – none32 Jan 30 '19 at 20:49
  • OK, the "alternative" meaning was not clear to me, but since you got a meaningful answer all good (when you see roughly 2 questions/month complaining, say, about low accuracy in *regression* settings, the assumption that the OP knows exactly what he/she is talking about starts feeling not that solid... :) – desertnaut Jan 30 '19 at 22:52

3 Answers3

19

If you are making a classifier, you need squared_hinge and regularizer, to get the complete SVM loss function as can be seen here. So you will also need to break your last layer to add regularization parameter before performing activation, I have added the code here.

These changes should give you the output

from keras.regularizers import l2
from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(64, activation='relu'))
model.add(Dense(1), kernel_regularizer=l2(0.01))
model.add(activation('softmax'))
model.compile(loss='squared_hinge',
              optimizer='adadelta',
              metrics=['accuracy'])
model.fit(X, Y_labels)

Also hinge is implemented in keras for binary classification, so if you are working on a binary classification model, use the code below.

from keras.regularizers import l2
from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(64, activation='relu'))
model.add(Dense(1), kernel_regularizer=l2(0.01))
model.add(activation('linear'))
model.compile(loss='hinge',
              optimizer='adadelta',
              metrics=['accuracy'])
model.fit(X, Y_labels)

If you cannot understand the article or have issues with the code, feel free to comment. I had this same issue a while back, and this GitHub thread helped me understand, maybe go through it too, some of the ideas here are directly from here https://github.com/keras-team/keras/issues/2588

Raheel Shahzad
  • 136
  • 2
  • 13
anand_v.singh
  • 2,768
  • 1
  • 16
  • 35
  • Thanks a lot, especially for the references, they've helped a lot, not just to get the ready-to-use solution, but to understand what is going under the hood. In my case it's a multi class classifier, so I'm using squared_hinge loss function. As far as I understood, the only difference between my code and that one you are provided is a utilization of regularizer and, by the way, this is the only part that I cannot understand now. I'll dig up more myself, because I'm not familiar with L2 regularizer at all. – none32 Jan 30 '19 at 20:45
  • can you explain why the last dense layer has only 1 node? – Chhaganlaal Feb 22 '20 at 07:13
  • 1
    Also explain W_regularizer as I am getting errors using that. – Chhaganlaal Feb 22 '20 at 07:30
  • When I try a similar code on mnist dataset, it gives very poor results like 10-11% accuracy. – Nitin1901 Sep 10 '20 at 03:13
  • @Chhaganlaal there is no parameter as such. You can use kernel_regularizer or bias_ instead – Nitin1901 Sep 10 '20 at 03:14
  • @NitinSai I wrote this answer around 1.5 years ago and I haven't done Machine learning for more than a year now, from the comments it appears that this answer is out of date, If I was to read ML again and fix this it would take a substantial time to relearn, if you are able to suggest an edit to update the answer do so and I will update the answer after verifying. – anand_v.singh Sep 10 '20 at 05:45
2

If you are using Keras 2.0 then you need to change the following lines of anand v sing's answer.

W_regularizer -> kernel_regularizer

Github link

model.add(Dense(nb_classes, kernel_regularizer=regularizers.l2(0.0001)))
model.add(Activation('linear'))
model.compile(loss='squared_hinge',
                      optimizer='adadelta', metrics=['accuracy'])

Or You can use follow

top_model = bottom_model.output
  top_model = Flatten()(top_model)
  top_model = Dropout(0.5)(top_model)
  top_model = Dense(64, activation='relu')(top_model)
  top_model = Dense(2, kernel_regularizer=l2(0.0001))(top_model)
  top_model = Activation('linear')(top_model)
  
  model = Model(bottom_model.input, top_model)
  model.compile(loss='squared_hinge',
                      optimizer='adadelta', metrics=['accuracy'])
Sivaram Rasathurai
  • 5,533
  • 3
  • 22
  • 45
  • do you know of any way of getting a confidence value or outputting probabilities? I detail this question here: https://stackoverflow.com/questions/67559912/svc-classifier-to-keras-cnn-with-probabilities-or-confidence-to-distinguish-untr – Apidcloud May 20 '21 at 14:01
0

You can use SVM with Keras implementation suing scikeras. It is a Scikit-Learn API wrapper for Keras. It was first release in May 2020. Below I have attached the official documentation link for it. I hope you will find your answer over there.

https://pypi.org/project/scikeras/#description

DANISH REZA
  • 1
  • 2