0

I'm seeing some strange things on a currently running experiment.

Google optimize reporting

  • The Observed Conversion Rate is noticeably higher on the Original, however Optimize's analysis is that the variant is more likely to be best. What should I trust here?
  • The Modeled Conversion Rate seems also way off, with a median CR of 1.2% (original) and 1.3% (variant). Why would the model overshoot reality so much?

Thanks for your guidance!

SpidrJeru
  • 119
  • 3
  • 13
  • This doesn't explain the results, but I had always believed Google (and other results that look like this with probability to be best) are based on a Bayesian stats. If you punch in your numbers on https://abtestguide.com/bayesian/, you find that "New Service Enabled" has only a 12% chance of being best, aligned with your intuition. I'm not sure what else Google is doing behind the scenes. – Timothy Chan Aug 05 '21 at 07:14
  • Exactly! And as always Google's model is a black box.. – SpidrJeru Aug 06 '21 at 11:31

0 Answers0