0

For those of you unfamiliar with Meka - it is an extension of Weka for multi-label classifiers. Meka and Weka are VERY similar, however, and so Weka users may be able to answer this question, too.

Basically, I want my results from my runs of various classifiers, and I want them all in a table so I can do Model Selection quickly (dynamically/automatically) without having to hardcode the values from each classifier, for the various evaluation metrics...

Is there a fool-proof, effective way to run multiple classifier experiments - say using cross validation - and get a table like the below:

Model                    Hamming_Loss    Exact_match    Jaccard    One_Error    Rank_Loss
Binary.Relevance           0.94             0.95          0.03      0.04       0.002
Classifier.Chains          0.91             0.94          0.06      0.04       0.03
Random.k-Labelsets         0.95             0.97          0.01      0.01       0.005
...                        ...              ...           ...       ...
...                        ...              ...           ...       ...
lrthistlethwaite
  • 494
  • 2
  • 6
  • 23
  • As a general strategy, can't you use meka from the command line and extract the output yourself using command-line tools for text processing? – knb Sep 13 '15 at 07:17

1 Answers1

0

Using Java you can manually create an array of different classifiers and iterate along it, saving the relevant output values in a matrix, for easy acces afterwards. You may even create a new dataset from the results obtained, for dynamic model selection, as you stated. But the key point is, as said, that you have to set up your classifiers array manually.

Classifiers[] cls = new Classifiers[clSize];
cls[0] = new J48();//or whatever you need
...

//one option:
cls[0].buildClassifier(dataset);
....

//another option
cls[0].crossValidateModel(dataset);
....

Hope to have helped. Comment, if you need further support.

shirowww
  • 533
  • 4
  • 18
  • I assume you download java source files that contain the class structure right? It's been a while since I coded in Java, but setting it up is always the hardest part. Do you know of instructions for setting up a the Java development framework on various operating systems (Windows, Unix)? – lrthistlethwaite Sep 15 '15 at 02:20
  • It's a piece of cake. Download from Oracle the latest version of the co-bundle JDK+Netbeans, install and start coding. Only thing you have to take into account is to add a new library with the WEKA jar, in the project options (right-click, etc). You could use also other IDEs, like Eclipse, telling it where the JDK is installed, through options menu. – shirowww Sep 15 '15 at 11:41