2

I've had an A/B Test running in Google Web Optimizer for six weeks now, and there's still no end in sight. Google is still saying: "We have not gathered enough data yet to show any significant results. When we collect more data we should be able to show you a winning combination."

Is there any way of telling how close Google is to making up its mind? (Does anyone know what algorithm does it use to decide if there's been any "high confidence winners"?)

According to the Google help documentation:

Sometimes we simply need more data to be able to reach a level of high confidence. A tested combination typically needs around 200 conversions for us to judge its performance with certainty.

But all of our conversions have over 200 conversations at the moment:

230 / 4061 (Original)
223 / 3937 (Variation 1)
205 / 3984 (Variation 2)
205 / 4007 (Variation 3)

How much longer is it going to have to run??

Thanks for any help.

Chuck Le Butt
  • 47,570
  • 62
  • 203
  • 289
  • I'm afriad i have no answer to your question, however a question for you if that's ok?!!! How have you found isomg GWO, would you recommend it? (ie easy to deploy? any perfomance issues?) sorry for interupting your question! – Haroldo Jun 11 '10 at 10:19
  • Yes. So far, so good. It's extremely easy to use and extremely powerful. A/B Testing isn't so much of an "if", it's more of a "when", and GWO seems like a good system -- if only I knew when the experiment would end! :) – Chuck Le Butt Jun 11 '10 at 10:29
  • I use this calculator to find the amount of data I'll need to get signficance (though, for distance, it uses 90% instead of 95%) http://www.usereffect.com/split-test-calculator So, for example, Original vs. Variation 2, you need 12,000 more results to get a conclusive answer. – Yahel Jul 16 '10 at 22:37

1 Answers1

2

Is there any way of telling how close Google is to making up its mind?

You can use the GWO calculator to help determine how long a test will take based on a number of assumptions that you provide. Keep in mind though that it is possible that there is not significant difference between your test combination, in which case a test to see which is best would take an infinite amount of time, because it is not possible to find a winner.

(Does anyone know what algorithm does it use to decide if there's been any "high confidence winners"?)

That is a mystery, but with most, if not all, statistical tests, there is what's called a p-value which is the probability of obtaining a result as extreme as the one observed by chance alone. GWO tests run until the p-value passes some threshold, probably 5%. To be more clear, GWO tests run until a combination is significantly better than the original combination, such that the result only has a 5% chance of occurring by chance alone.

For your test there appears to be no significant winner, it's a tie.

erikvold
  • 15,988
  • 11
  • 54
  • 98
  • I can't believe it would keep going forever :( Thanks for the information, Erik! – Chuck Le Butt Jun 14 '10 at 09:54
  • It's interesting to note that this page talks about "inconclusive results", possibly suggesting that tests CAN end without a clear winner? What do you think? http://www.google.com/support/websiteoptimizer/bin/answer.py?hl=en_UK&ctx=gwo-sal-%2Fsiteopt%2Freports&answer=55947 – Chuck Le Butt Jun 14 '10 at 10:18
  • "possibly suggesting that tests CAN end without a clear winner?" no it's saying what I'm saying, which is that the test can go on forever until you do something about it. – erikvold Jun 14 '10 at 11:21
  • In my experience, you're right, 5% is when they're ready to declare a winner. – Yahel Jul 16 '10 at 22:34