0

Assume I have a website with YES and NO buttons. Now, I log number of YES clicks and number of NO clicks for two days.

After having this data, I want to do A/B-test analysis on these data. So, I split data into 50-50 split. Use one as Control data and one for Test data. For Test data I count number of Yes and No clicks in 10 hours (instead of two days).

Do you think it is a good way to do A/B-test? As far as I know, I need two versions of website with slightly different feature while in my case website is identical for all users and I only split original data with some assumption. If it is not right, please refer me to an article or website.

user2274644
  • 3
  • 2
  • Or you can use a single web site with a state machine built into it to select which page is displayed using a randomizer or criteria of your choosing. – duffymo Apr 12 '13 at 13:52
  • However, the website doesn't change for both group. I am using the same website (without any change) and only split data after gathering. – user2274644 Apr 12 '13 at 13:57

1 Answers1

0

You need the two groups of users to be identical (selected randomly) and so it is much preferred to run the groups in parallel.

For more info on how to interpret the results I highly recommend this blog: http://visualwebsiteoptimizer.com/split-testing-blog/, especially "What you really need to know about mathematics of A/B split testing" and "How large should your A/B test sample size be?" are must reads.

There is also a nice significance calculator available here.

sashkello
  • 17,306
  • 24
  • 81
  • 109
  • However, the website doesn't change for both groups. I am using the same website (without any change) and only split data after gathering. This looks for me not as ABtest. as in ABtest you should have two websites with slightly different specification and run in parallel. – user2274644 Apr 12 '13 at 13:58
  • What are you testing then if everything is the same? – sashkello Apr 12 '13 at 14:00
  • Gather data for two days from one website. Split gathered data into two groups: Control group is percentage of number of YES clicks in 2 days and Test group is percentage of number of YES clicks in 10 hours. – user2274644 Apr 12 '13 at 14:04
  • Doing like this will detect if in those 10 hours the click rate was different, that's all. This is not A/B test at all (although same math applies to interpreting the stats). Again, can you let me know what are you trying to do with this? As I see it, you have different kind of users in that 2-day timeframe and 10-hour timeframe and want to find a difference (sort of A/B test on users, rather than webpage). – sashkello Apr 12 '13 at 14:06
  • Exactly. This is what I think. It only shows what is the difference between click rate for 10 hours and 2 days not ABTest. However, I don't have any article/refrence to prove them they are wrong. I still don't know exactly what they are looking for, it seems that they want to put some time limitation on incoming ads in their website. – user2274644 Apr 12 '13 at 14:13
  • There is nothing wrong with it, it is not A/B test but as I said very similar thing. As I don't know what is the actual thing you try to verify, I can't be more explicit. The only thing I must say is that you should be sure to have those time intervals identical as well (comparing 2-day stats with 10 hours at night or during the daytime will certainly give different outcomes and not a proper set up) – sashkello Apr 12 '13 at 14:16