4

In the spacy's text classification train_textcat example, there are two labels specified Positive and Negative. Hence the cats score is represented as

cats = [{"POSITIVE": bool(y), "NEGATIVE": not bool(y)} for y in labels]

I am working with Multilabel classfication which means i have more than two labels to tag in one text. I have added my labels as

textcat.add_label("CONSTRUCTION")

and to specify cats score I have used

cats = [{"POSITIVE": bool(y), "NEGATIVE": not bool(y)} for y in labels]

I am pretty sure this is not correct. Any suggestions how to specify the scores for cats in multilabel classification and how to train multilabel classification? Does the example from spacy works for multilabel classification too?

Subha Maharjan
  • 85
  • 2
  • 7

2 Answers2

3

If I understood you correctly, you have a list of categories, and your data can have multiple categories at once. In that case you cannot use "POSITIVE": bool(y), "NEGATIVE": not bool(y) to mark your classes. Instead, try writing a function which will return a dictionary with categories based on the classes. For example, consider having a following list of categories: categories = ['POLITICS', 'ECONOMY', 'SPORT']. Now, you can iterate over you train data, calling a function for each training example.

This function can look like this:

def func(categories):
    cats = {'POLITICS': 0, 'ECONOMY': 0, 'SPORT': 0}
    for category in categories:
        cats[category] = 1
    return {'cats': cats}

Having a training example with two categories (for example POLITICS and ECONOMY), you can call this function with a list of categories (labels = func(['POLITICS', 'ECONOMY']) and you will get a full dictionary with classes for this example

Oleg Ivanytskyi
  • 959
  • 2
  • 12
  • 28
1

The example scripts are mainly quick demos for a single use case and you're right that this isn't the right kind of evaluation for a multilabel case.

The underlying spacy Scorer and the spacy evaluate CLI (https://spacy.io/api/cli#evaluate) report the macro-averaged AUC ROC score for multilabel classification.

You can use the Scorer with nlp.evaluate() (https://spacy.io/api/language#evaluate) or through spacy evaluate / spacy train.

If your data is in the simple TRAIN_DATA format from the example script, nlp.evaluate() is probably the easiest way to run the Scorer, since spacy evaluate would require you to convert your data to spacy's internal JSON training format.

The model settings (specified when you initialize the pipeline component) are used to pick an appropriate evaluation metric (obviously these aren't the only possible metrics, just one suitable metric for each configuration):

  • f-score on positive label for binary exclusive,
  • macro-averaged f-score for 3+ exclusive,
  • macro-averaged AUC ROC score for multilabel
aab
  • 10,858
  • 22
  • 38