0

Recently we launched a feature on one of our website pages.I have all the historical data worth 6 months about the page including impressions and CTR .It has been 20 days since we launched the feature and now we want to know if there is any significant lift in CTR post-launch. Is there a way to determine how many impressions is statistically significant to conduct a T-test to check lift in CTR assuming page views before launch was control group and post launch is test group. How much historical data do I need to look at for control group and evaluate the required sample size for test group based on that. Any lead OR different approach is highly appreciated.

M_Nayak
  • 1
  • 3
  • Your terminology is confusing. What is CTR? What do you mean by significant __lift__? What are impressions?Are you trying to compare the mean (of something?) before and after this new feature (I am assuming because you mentioned t-tests)? What is this feature you are measuring and what type is it? 6 months should be more than enough if we are talking internet page views. – user2974951 Nov 29 '18 at 09:11
  • Sorry for the confusion.CTR is click through rate. Impressions mean pageviews and clicks are the number of clicks to various links on the web page. We want to know if there is a statistically significant change in our CTR before and after feature launch. What I do not know is how to decide what is the sample size for my control and test group with confidence interval of 95%. In a A/B test, you show both variants to equal number of users at the same time. But in this case its before and after launch and thats where it gets complicated. – M_Nayak Nov 29 '18 at 14:21

0 Answers0