-1

My boss asked me to perform a T-Test to test the significance for a certain metric we use called conversion rate.

I have collected 18 months worth of data for this metric dating April 1, 2017 - September 30th, 2018.

He initially told me to collect 12 - 14 months of the data and run a t-test to to look for significance of the metric. (Higher conversion rate means better!).

I'm not really sure how to go about it. Do I split the data up into 9 month samples i.e. Sample 1: April 2017 - December 2017, Sample 2: January 2018 - September 2018 and run a two sample t-test? Or would it make sense to compare all of the data against a mean like 0?

Is there a better approach to this? The bottom line is he wants to see that the conversion rate has significantly increased over time.

Thanks, - Keith

Keith
  • 103
  • 1
  • 9

1 Answers1

0

My advice is to dump the t-test and look only at the magnitude of the change in the conversion rate. After all, the conversion rate is what's important to your business. By the way, looking at the magnitude of something practically relevant is called "effect size analysis"; a web search for that should turn up a lot of resources. To get started, just make a plot of the available data -- is conversion rate going up or going down or what?

Further questions should be directed to stats.stackexchange.com instead of SO. Good luck and have fun.

Robert Dodier
  • 16,905
  • 2
  • 31
  • 48
  • thank you so much! So i had a bit of a debate with my boss over this - the conversion rate is a ratio of people who rent (a car) to people who register for the service but do not rent a car. every month the population size changes i.e Month 1 = 10 rent & 100 register (0.1 conversion rate) Month 2 = 20/160 (0.125 conversion rate). I argued that we cannot compare these ratios because their bases vary every month, but should instead use a t-test to compare the change in the rate. So if i have 12 months worth of data and I want to see if the 12 data point is sig. i would take the mean change – Keith Oct 31 '18 at 13:31
  • for the 11 months and then see how far away from the mean the 12th month change would be. Is this a sound understanding? Thank you! – Keith Oct 31 '18 at 13:32
  • I agree that the month to month ratios are not exactly comparable, due to the differing denominators, but I think that's probably a second-order effect. My advice is to first plot all three of the available numbers over time: number registered, number rented, ratio. Look for trends or cycles there, and try to relate that to any known events (e.g. advertising campaign, regulatory or political events, festivals or other events that bring in a lot of people). Is the ratio really the most important? Maybe it's the actual number of rentals since that's how you get paid. – Robert Dodier Oct 31 '18 at 19:12
  • But anyway looking at the simplest plots or tables will get the discussion going about what's really important for your business. A little math goes a long way, if you know where to apply it. – Robert Dodier Oct 31 '18 at 19:13