-1

I have a multi-class classification problem with the classes X-Small, Small, Medium and Big.

I have the following requirement:

enter image description here

Summary:

  • Predicting a X-Small as X-Small is GOOD
  • Predicting a X-Small as Small is OK
  • Predicting a X-Small as Medium is BAD
  • Predicting a X-Small as Big is BAD

Similarly

  • Predicting a Small as X-Small is OK
  • Predicting a Small as Small is GOOD
  • Predicting a Small as Medium is OK
  • Predicting a Small as Big is BAD

Similarly

  • Predicting a Medium as X-Small is BAD
  • Predicting a Medium as Small is OK
  • Predicting a Medium as Medium is GOOD
  • Predicting a Medium as Big is OK

Similarly

  • Predicting a Big as X-Small is BAD
  • Predicting a Big as Small is BAD
  • Predicting a Big as Medium is OK
  • Predicting a Big as Big is GOOD

Question

  • What is the ideal classification metric to use?
  • What is the ideal loss function to use?
Kartheek Palepu
  • 972
  • 8
  • 29
  • This is not a *programming* question, hence it is off-topic here; please see the intro and NOTE in https://stackoverflow.com/tags/machine-learning/info – desertnaut Sep 15 '22 at 09:40
  • @desertnaut: I actually posted the question in data science stack: https://datascience.stackexchange.com/questions/114423/weighted-classification-metric-for-multi-class-classification and I didn't receive any response (till now). That community is not super active. – Kartheek Palepu Sep 15 '22 at 18:52

1 Answers1

1

There are different alternatives, which are basically based upon binary 2x2 contingency tables: For multiclass classification, the easy way to go is to use:

  • accuracy = sum(diag(matrix))/sum(matrix). The closest to one the better.
  • But you can also use it for each class:
    • Re-label to make a 2x2 contingency table, like the class of interest against the rest, and calculate every available index, sensibility, specificity, accuracy, etc. To create the intra-class metrics.
    • Then create the global metric by the mean of the metric by the number of classes. This is usually called the macro-sensibility, macro-xxxx measurement.
Cristobal Fresno
  • 11
  • 2
  • Thanks for the response. What you answered is a typical multi-class classification metric. What I need is much more critical penalization based on prediction extremes. i.e. In your case - I believe: Predicting `Small` as `Medium` and `Small` as `Big` will result in same score. – Kartheek Palepu Sep 14 '22 at 18:24
  • 1
    Ah, you can first multiply your confusion matrix by a weighted matrix where you have the principal diagonal is 1 (good), then x.x for OK and y.y for Bad, before calculating the metrics. So, depending on the x.x and y.y values you will add more/less weight to what you are looking for – Cristobal Fresno Sep 14 '22 at 18:47
  • I just tried it with weights 1, -1, -2 and it is working. Please add your comment to your answer so that I can upvote and accept it as **Answered**. – Kartheek Palepu Sep 14 '22 at 19:18