Why is Google Experiments showing significant conversion differences in my A/A test?

Question

I have implemented an A/B test using Google Experiments. However, we did not want the A/B test to start for a period of time and therefore put a date driven condition on showing the B variant. Out of curiosity I started the experiment anyway before the B variant go live date. Hence, at the start of the experiment, Google Experiments was choosing a variation for me (A or B, see code below) and regardless of the chosen variation, I would show variant A (since we hadn't hit the go live date yet). This effectively gives me an A/A test which I expected to be a dead heat. Google doesn't know that I'm showing the exact same page elements to each variant at this stage.

However, I get the following results (showing page views per session):

The above image shows the results up to the day that the B variant went live. While I expected some natural variation in the page views, variation B is so consistently winning, that something seems wrong as it is showing an identical page to variation A at this point. Any ideas what I might be doing wrong?

Here's the code we use:

      <script type="text/javascript" 
              src="//www.google-analytics.com/cx/api.js?experiment=MY_EXPERIMENT_ID"><!-- --></script>

      <script>
            function enableVariationB(shouldEnable) {
                var enableExperimentDate = new Date(2015, 6, 10);    
                var today = new Date();

                if (today < enableExperimentDate || shouldEnable) {
                    $('#variation_B').show();
                } else {
                    $('#variation_B').hide();
                }
            }

            //Call to Google Experiments to return 1 or 2 depending on 
            //which variation is to be shown to the user
            var chosenVariation = cxApi.chooseVariation();

            var pageVariations = [
                      function() {  
                          enableVariationB(true);  
                      }, 
                      function() {
                          enableVariationB(false);    
                      }
            ];

            $(document).ready(
                 pageVariations[chosenVariation]
            );
     </script>

Answer 1

Hmm. Before looking into the data, in the experiment setup, the today < enableExperimentDate check should be done before calling cxApi.chooseVariation().

Reason: Even if a visitor is assigned to variation B from Google's point-of-view, if s/he failed the experiment date check, what a user would see is variation A even if s/he is supposed to be in variation B.

Now, looking at the data,

1. Not enough traffic: How much traffic does your website/application get? It could be just coincident if the traffic volume is low, and the result is not conclusive.

2. Unclear success metrics/overall evaluation criteria (OEC): Even though variation B has a higher page view count than variation A, what success metric is your experiment trying to optimize? Looking at just the view count is not sufficient.

3. Drawing conclusion too early: If the OEC is, for example, increase page view from 1 to 2, based on the traffic volume your site get per day, and let's say it takes 1 month to get that amount of data to reach a 95% confidence on your experiment, experimenter should not look at the result early and conclude that which treatment won or not -- Because it does not have sufficient traffic yet to give you the 95% confidence.

4. It's still positive for a false positive: Even if we are confident about the result with 95% confidence, there is still a 5% chance of running into false positive result -- Meaning, if you run 20 tests, statistically speaking, 1 of the test result will be incorrect. That's why it is not uncommon for experimenters to re-run experiments at a later time just to be sure.

Hope the points above help!