0

I have two changes on my web page but I'm monitoring a bunch of variables. So what I'm able to extract from my website monitoring experiment is as follows:

Original solution: Visitors,   body link click-visitors,    most popular click-visitors,    share-visitors.
Solution with some change: Visitors,   body link click-visitors,    most popular click-visitors,    share-visitors.

I was wondering about simple 2 sample portion test. Take each of the monitored variable and compute portion test for original and changed solution.

I don't know if it tells me something about the overall result - if original solution is better than the solution with some change or not.

Is there something better what can I use for this purpose. I'll appreciate any of your advice.

martinkabe
  • 1,079
  • 2
  • 12
  • 27

1 Answers1

0

Sounds to me like you’re confusing two things: business metric of interest and test for statistical significance. The former is some business mesurement that you would like to measure for. This could be sales, conversion, subscription rate, or many others. See e.g. this paper for a good discussion on the perils of using the wrong metric. Statistical significance is a test that tells you if the number of measurements you’ve seen so far is enough to substantiate a claim that the difference between the two experiences is very unlikely random. See e.g. this paper for a good discussion.

Igor Urisman
  • 717
  • 1
  • 6
  • 22