Need of bonferroni correction in A/B testing

Question

I am a newbie in the field of Data Science. I came across the below statements which read:

1. More metrics we choose in our A/B testing, higher the chance of getting significant difference by chance.

2. To eliminate this problem we use Bonferroni correction method.

What does the 1st statement mean? How does it increase the chances of getting false positives? and how does the Bonferroni correction method help us here?

Answer 1

With p value of 0.05 (which is a commonly used level of statistical significance), you will get false positive results 5% of time. Thus if in your analysis you have one test, your chance of false positive is 5%. If you have two tests, you´ll have 5% for the first AND 5% for the second. Et cetera.

So for each additional test, your risk increases. Still, as you want to keep your total risk level at 0.05, you either set more strict level of statistical significance (smaller p value), or use some statistical method to correct for multiple comparisons. Bonferroni correction is one of such methods.