How can I calculate the Failed Acceptance Rate and False Recognition Rate?

Question

How can I calculate the FAR and FRR for an authentication device? (Which I assume is using Biometrics). Otherwise is their any statistics on the subject that is easily accessible?

Answer 1

A biometric system can work in two modes, which must be distinguished during evaluation: verification and identification.

In verification mode, a user presents his identity and the biometric device verifies that the identity matches (imagine that you go to customs and you present your ID with face image, the officer verifies your real face and the image on ID).

In identification mode, no assumption of identity is made in the beginning and comparison to all templates has to be made (for example, a killer left his fingerprint at crime scene and then police identifies him using his fingerprints -- no assumption is made, they just use the acquired fingerprint and they compare it to all their fingerprints in their database).

It is therefore necessary to distinguish between these two situations, because identification is generally more demanding.

Lets assume, that you are evaluating a verification mode of biometric system. Imagine a biometric system, which assigns all authentication attempts a score from interval [0, 1]. 0 means no match and 1 means full match. Obviously if you set your threshold to 0, all genuine users are admitted, but all impostors are admitted also. On the other hand, if you set your threshold to 1, no one is admitted. So for real usage, you usually set the threshold somewhere between. This might cause, that not all genuine users are admitted and some impostors are admitted. As you can see, there are two error rates: FAR (False Accept Rate) and FRR (False Reject Rate).

FAR is calculated as a fraction of impostor scores exceeding your threshold.

FRR is calculated as a fraction of genuine scores falling bellow your threshold.

Example: We have a fingerprint system. In order to evaluate a performance of any biometric system, we need to gather a database. Assume, that we have done that and the database consists of 10 legitimate users (USER_1-USER_10) and each user provided his finger 10 times (10x10 = 100 images in total). Lets assume, that single image is sufficient for template creation. You select a user (e. g. USER_1) and one his fingerprint image and create the template. The rest of his images you use to verify the fingerprint and you receive 9 genuine scores. All images of other users you use as a impostors and you receive 90 impostor scores. You repeat the template generation for all images and all users and in total you receive 900 genuine scores and 9000 impostor scores. These scores are usually used to generate so called ROC curves to choose the best threshold suting your problem. If you have chosen a threshold, you can calculate the FAR and FRR using the definition I stated above.

Lets assume, that we have chosen as threshold 0.7 and 100 impostor scores exceed the threshold and 50 genuine scores fall below the threshold.

So FAR = 1.1% and FRR = 5.6%.

Answer 2

The FAR and FRR are always calculated specific to a particular population. This is one of the downsides to using a FAR and FRR to measure performance unless you have a consistent population you are comparing against.

For example, if my population consists of me and two other people the odds of me having a false accept are very low, likely 0% and my false reject rate, provided that I am able to reliable capture usable samples is also going to be very low, likely 0%, which would be extremely impressive without the context of the database size.

To accurately test the system, I would recommend you find out what the standards are for that particular type of authentication. For example, in fingerprint matching NIST has a database of images that they use to measure different algorithms. There are a number of algorithms that have all been measured against that database and so you would be comparing apples to apples. If you do not have the same database across algorithms then you are comparing apples to oranges if that makes sense.

Answer 3

if you were looking for a tools or an automation software to do your work ,you can go for the matlab with RT-tool software .

mathworks.com provides good source content which is easily understandable and easy to modify

Answer 4

FAR = FP / (FP + TN)
FRR = FN / (FN + TP)

Where:

• FAR: False Acceptance Rate
• FRR: False Rejection Rate
• TP: True Positive
• FP: False Positive
• TN: True Negative
• FN: False Negative

Answer 5

For calculate FAR/FRR curve you can use:

def curve_frr_far(
    targets, #list of 0 and 1
    genuine_probabilities,
    genuine_label = 0 # 0 or 1
):
    sort_idxs = genuine_probabilities.argsort()
    targets = targets[sort_idxs]
    genuine_probabilities = genuine_probabilities[sort_idxs]
    
    far = []
    frr = []
    thresholds = []

    len_genuines = (targets == genuine_label).sum() 
    len_spoofs = (targets != genuine_label).sum()
    
    len_targets = len(targets)
    th = 0.0
    fa = len_spoofs
    fr = 0

    thresholds.append(th)
    far.append(fa/len_spoofs)
    frr.append(fr/len_genuines)
        
    for index in range(len_targets):
        genuine_p = genuine_probabilities[index]
        label = targets[index]

        if label == genuine_label:
            fr += 1
        else:
            fa -= 1

        if genuine_p != th:
            th = genuine_p
            far.append(fa/len_spoofs)
            frr.append(fr/len_genuines)
            thresholds.append(th)

    if th != 1.0:
        th = 1.0
        fa = 0
        fr = len_genuines
        far.append(fa/len_spoofs)
        frr.append(fr/len_genuines)
        thresholds.append(th)
    
    return thresholds, frr, far