0

I am running an AAB test, which means there are two control groups in the test to make sure the population distribution is correct. And the results tell me that the duplicated control group show positive results than the original group.

I duplicated the test multiple times, on two applications, but the results between original groups and the duplicated control groups are still varied.

Is there any possible reason for this?

Long Quach
  • 1
  • 1
  • There's not enough information here to say anything concrete. If you'd like to report a bug, [file a bug report](https://firebase.google.com/support/contact/bugs-features/). If you'd like an answer why you see a certain behavior, show us enough information on what you've done and what you see. – Frank van Puffelen Dec 13 '18 at 14:50

1 Answers1

0

Some percent of online field experiments between identical control groups will show statistical winners. This is called a false positive. Firebase support covers this topic.

Different statistical frameworks have different ways of managing error rate (and false positive rate), eg:

Leanplum

Optimizely

If you experience a false positive multiple times, this is statistically unlikely and usually points to an implementation error or a software error.

Disclaimer: I work at Optimizely.

user3473097
  • 1
  • 2