5

I am using the library e1071. In particular, I'm using the svm function. My dataset has 270 fields and 800,000 rows. I've been running this program for 24+ hours now, and I have no idea if it's hung or still running properly. The command I issued was:

svmmodel <- svm(V260 ~ ., data=traindata);

I'm using windows, and using the task manager, the status of Rgui.exe is "Not Responding". Did R crash already? Are there any other tips / tricks to better gauge to see what's happening inside R or the SVM learning process?

If it helps, here are some additional things I noticed using resource monitor (in windows):

  • CPU usage is at 13% (stable)
  • Number of threads is at 3 (stable)
  • Memory usage is at 10,505.9 MB +/- 1 MB (fluctuates)

As I'm writing this thread, I also see "similar questions" and am clicking on them. It seems that SVM training is quadratic or cubic. But still, after 24+ hours, if it's reasonable to wait, I will wait, but if not, I will have to eliminate SVM as a viable predictive model.

gung - Reinstate Monica
  • 11,583
  • 7
  • 60
  • 79
Jane Wayne
  • 8,205
  • 17
  • 75
  • 120
  • 1
    Please try to use standard English grammar, capitalization, & punctuation, etc., in writing your questions. It will make it easier for people to try to help you. – gung - Reinstate Monica Mar 31 '14 at 20:54
  • 1
    hopefully you will get help from experts but failing that my generic approach is to get the tool working for a smaller dataset, then try larger and larger ones, noting by how much the time is increasing (if necessary get enough data points to plot a graph). – TooTone Mar 31 '14 at 20:55

2 Answers2

4

As mentioned in the answer to this question, "SVM training can be arbitrary long" depending on the parameters selected.

If I remember correctly from my ML class, running time is roughly proportional to the square of the number training examples, so for 800k examples you probably do not want to wait.

Also, as an anecdote, I once ran e1071 in R for more than two days on a smaller data set than yours. It eventually completed, but the training took too long for my needs.

Keep in mind that most ML algorithms, including SVM, will usually not achieve the desired result out of the box. Therefore, when you are thinking about how fast you need it to run, keep in mind that you will have to pay the running time every time you tweak a tuning parameter. Of course you can reduce this running time by sampling down to a smaller training set, with the understanding that you will be learning from less data.

Community
  • 1
  • 1
merlin2011
  • 71,677
  • 44
  • 195
  • 329
  • 1
    This is an extremely informative answer, but it doesn't exactly explain how to tell if the library or R session has crashed. – bright-star May 07 '14 at 01:55
  • @TrevorAlexander, That is a fair criticism, but I addressed the question of "how long is it reasonable to wait", so this does address the question, although not ALL of it. – merlin2011 May 07 '14 at 02:03
1

By default the function "svm" from e1071 uses radial basis kernel which makes svm induction computationally expensive. You might want to consider using a linear kernel (argument kernel="linear") or use a specialized library like LiblineaR built for large datasets. But your dataset is really large and if linear kernel does not do the trick then as suggested by others you can use a subset of your data to generate the model.

Krrr
  • 452
  • 1
  • 3
  • 15