1

I'm a student studying with a focus on machine learning, and I'm interested in authentication. I am interested in your library because I want to calculate the EER.

Sorry for the basic question, but please tell me about bob.measure.load.split().

Is the file format required by this correct in the perception that the first column is the correct label and the second column is the predicted score of the model?

like

# file.txt    
|label|prob |
| -1  | 0.3 |
| 1   | 0.5 |
| -1  | 0.8 |

...

In addition, to actually calculate the EER, should I follow the following procedure?

neg, pos = bob.measure.load.split('file.txt')
eer = bob.measure.eer(neg, pos)

Sincerely.

Hirotaka Kawashima
  • 13
  • 3

1 Answers1

0

You have two options of calculating EER with bob.measure:

  1. Use the Python API to calculate EER using numpy arrays.
  2. Use the command line application to generate error rates (including EER) and plots

Using Python API

First, you need to load the scores into memory and split them into positive and negative scores. For examples:

import numpy as np
import bob.measure
positives = np.array([0.5, 0.5, 0.6, 0.7, 0.2])
negatives = np.array([0.0, 0.0, 0.6, 0.2, 0.2])
eer = bob.measure.eer(negatives, positives)
print(eer)

This will print 0.2. All you need to take care is that your positive comparison scores are higher than negative comparisons. That is your model should score higher for positive samples.

Using command line

bob.measure also comes with a suite of command line commands that can help you get the error rates. To use the command line, you need to save the scores in a text file. This file is made of two columns where columns are separated by space. For example the score file for the same example would be:

$ cat scores.txt
1 0.5
1 0.5
1 0.6
1 0.7
1 0.2
-1 0.0
-1 0.0
-1 0.6
-1 0.2
-1 0.2

and then you would call

$ bob measure metrics scores.txt
[Min. criterion: EER ] Threshold on Development set `scores.txt`: 3.500000e-01
================================  =============
..                                Development
================================  =============
False Positive Rate               20.0% (1/5)
False Negative Rate               20.0% (1/5)
Precision                         0.8
Recall                            0.8
F1-score                          0.8
Area Under ROC Curve              0.8
Area Under ROC Curve (log scale)  0.7
================================  =============

Ok it didn't print EER exactly but EER = (FPR+FNR)/2.

Using bob.bio.base command line

If your scores are the results of a biometrics experiment, then you want to save your scores in the 4 or 5 column formats of bob.bio.base. See an example in https://gitlab.idiap.ch/bob/bob.bio.base/-/blob/3efccd3b637ee73ec68ed0ac5fde2667a943bd6e/bob/bio/base/test/data/dev-4col.txt and documentation in https://www.idiap.ch/software/bob/docs/bob/bob.bio.base/stable/experiments.html#evaluating-experiments Then, you would call bob bio metrics scores-4-col.txt to get biometrics related metrics.

183.amir
  • 422
  • 3
  • 25