0

How does selection_rate_group_summary decide the cutoff value when we pass the output of a mitigated model to it in fairlearn module.

The selection rate in one of our data is really high all the time while using the function which is unnatural.

Anindya Sankar Dey
  • 101
  • 13

1 Answers1

0

There is no cutoff value. selection_rate_group_summary simply collects selection rates for all groups and overall.

Example:

>>> from fairlearn.metrics import selection_rate_group_summary
>>> y = [0, 1, 1, 1, 1, 1, 1, 1]
>>> sensitive_features = ["a", "b", "a", "b", "a", "b", "a", "b"]
>>> selection_rate_group_summary(y, y, sensitive_features=sensitive_features)
{'overall': 0.875, 'by_group': {'a': 0.75, 'b': 1.0}}

Similarly, if we have lots and lots of samples and the numbers get very large, this still works (no cutoff):

>>> y = [1] * 1000000
>>> y[0] = 0
>>> sensitive_features = ["a", "b"] * 500000
>>> selection_rate_group_summary(y, y, sensitive_features=sensitive_features)
{'overall': 0.999999, 'by_group': {'a': 0.999998, 'b': 1.0}}

If you meant something else please update the question or write a message to the community on Gitter

Roman Lutz
  • 48
  • 7
  • Well Gitter is blocked by my company for suspected malware, this is the only solution. What I meant was, how does the mitigated model chose 1/0, now in know the answer looking at code – Anindya Sankar Dey Oct 27 '20 at 19:04
  • If you meant the Gitter desktop app, there's also a website you can open in the browser. Let me know if anything else is unclear. I'm not sure this helped... – Roman Lutz Oct 31 '20 at 00:40