Multi-class Fairness Metric Implementation

Question

FairGBM is a fairness-aware ML model built upon LightGBM. You pass in your training data matrix (X) and training labels vector (Y), like you would normally do any gradient boosting algorithm and fit the model.

However, if you want the model to be "fair", you can specify a sensitive attribute (S) which is typically categorical/has groups in nature. Think about the gender column in the titanic dataset (male and female groups).

The current implementation of their fairness function is limited to binary classification:

def compute_fairness_ratio(y_true: np.ndarray, y_pred: np.ndarray, s_true, metric: str) -> float:
    """Compute fairness metric as the disparity (group-wise ratio)
    of a given performance metric.
    Parameters
    ----------
    y_true : np.ndarray
        The true labels.
    y_pred : np.ndarray
        The binarized predictions.
    s_true : np.ndarray
        The sensitive attribute column.
    metric : str
        The performance metric used to compute disparity.
    Returns
    -------
    value : float
        The fairness metric value (between 0 and 1).
    """
    metric = metric.lower()
    valid_perf_metrics = ("fpr", "fnr", "tpr", "tnr")
    
    def compute_metric(y_true, y_pred):
        tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
        
        if metric == "fpr":
            return fp / (fp + tn)
        elif metric == "tnr":
            return tn / (fp + tn)
        elif metric == "fnr":
            return fn / (fn + tp)
        elif metric == "tpr":
            return tp / (fn + tp)
        else:
            raise ValueError(f"Invalid metric chosen; must be one of {valid_perf_metrics}; got '{metric}'")

    groupwise_metrics = []
    for group in pd.Series(s_true).unique():
        group_filter = (s_true == group)
        
        groupwise_metrics.append(compute_metric(
            y_true[group_filter],
            y_pred[group_filter],
        ))

    return min(groupwise_metrics) / max(groupwise_metrics)

However, I want to generalize this to multi-class (the data I am working with has 3 classes). So far what I have tried is shown below, but the output does not make sense. How would you reimplement this function so that it is suited for multi-class classification? More on FairGBM here.

def compute_class_wise_metrics(Y_test, Y_pred):

    unique_labels = np.unique(np.concatenate((Y_test, Y_pred), axis=None))
    if len(unique_labels)==1:
      unique_labels = np.concatenate((unique_labels, np.zeros(1)))

    TN, FP, FN, TP = confusion_matrix(Y_test, Y_pred, labels=unique_labels).ravel()

    return {
        "fpr": FP / (FP + TN + 1e-6),
        "fnr": FN / (FN + TP + 1e-6),
        "tpr": TP / (FN + TP + 1e-6),
        "tnr": TN / (FP + TN + 1e-6),
    }
def compute_fairness_ratio(Y_test, Y_pred, S_true, metric):
    metric = metric.lower()
    if metric not in ['fpr', 'tpr', 'fnr', 'tnr']:
        raise ValueError(f"Unsupported metric type.")

    groupwise_metrics = []
    for group in np.unique(S_true):

        group_test, group_pred = Y_test[S_true == group].values, Y_pred[S_true == group]
        cw_metric = compute_class_wise_metrics(group_test, group_pred)[metric]
        class_counts = np.zeros(len(np.unique(Y_test)))
        for class_, count in pd.Series(group_test).value_counts().items():
            class_counts[int(class_)] = count
        groupwise_metrics.append(cw_metric)

    groupwise_metrics = list(set(groupwise_metrics))
    groupwise_metrics.remove(0.0)

    return min(groupwise_metrics) / max(groupwise_metrics)

Also, find here an example notebook where they run it on a binary classification problem.

Multi-class Fairness Metric Implementation

0 Answers0