We are using Google Experiments heavily through the experiments api and recently started seeing experiments ending with the winner having lower score than other variations -- see the attached screenshots of two recent experiments.
Anyone else seen that? Do you have any idea what does this mean?
Probably because confidence it's really low, so Google it's not confident to apply any variation.
- Choosing the winner of your experiment
Over time, you should see either the original ad setting or the variation outperform the other. To help you decide the right time to choose your winner, we calculate a confidence score for the two settings. This score indicates how likely the original or the variation is to be the better performing (e.g., the highest earning) setting.
We recommend that you wait until one of the scores reaches at least 95% before you choose that setting as the winner. When a setting has a 95% confidence score it means that we are 95% certain that that setting is the better performing of the two.
You can read more:
https://support.google.com/adsense/answer/6321879?hl=en
I suggest also this page: https://support.google.com/adsense/answer/6321909
When you're choosing the winner of your experiment, we encourage you to consider the Overall Change as well as the revenue (RPM) and the relative Quality Score. The Overall Change equals the change in RPM plus the change in Quality Score of the variation. In general, a positive Overall Change will result in better long-term performance, even when either the RPM change or the Quality Score change is negative. In situations where the RPM and the Quality Score point to different winners, a positive Overall Change can help you to make a final decision.
