power.prop.test function not interchangeable

Question

I am using the power.prop.test function in R.

I am doing an A/B Test where I am determining the lift from a minimum amount of impressions per group in order for the A/B Test to be significant.

When I run the function like below, I get the second proportion (p2) to be 0.0001870215 with n being 2,571,429 per group:

original_conversion_rate<-0.00009

power.prop.test(n=2571429,
  p1=original_conversion_rate, 
            power=0.8, 
            sig.level=0.05)

My answer is this:

     Two-sample comparison of proportions power calculation 

          n = 2571429
         p1 = 9e-05
         p2 = 0.0001870215
  sig.level = 0.05
      power = 0.8
alternative = two.sided

NOTE: n is number in *each* group

When I rerun this with my answer for p2 (0.0001870215) to solve for n, a different n comes up (230,952.6):

original_conversion_rate<-0.00009

power.prop.test(
  p1=original_conversion_rate, 
  p2=0.0001870215,
            power=0.8, 
            sig.level=0.05)

My n changes to this:

 Two-sample comparison of proportions power calculation 

          n = 230952.6
         p1 = 9e-05
         p2 = 0.0001870215
  sig.level = 0.05
      power = 0.8
alternative = two.sided

Why would n change in this case?

score 1 · Accepted Answer · answered Feb 21 '18 at 18:21

Your frequencies are so small that miniscule changes will have a huge impact - especially with the default tolerance level used in power.prop.test.

If you take one iteration more you get

power.prop.test(n=230952.5,
  p1=original_conversion_rate, 
            power=0.8, 
            sig.level=0.05)

     Two-sample comparison of proportions power calculation 

              n = 230952.5
             p1 = 9e-05
             p2 = 0.0001870215
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

which is the same value for p2 as before. Now, if we go back to your first calculation and lower the tolerance we get

power.prop.test(n=2571429,
  p1=original_conversion_rate, 
            power=0.8, 
            sig.level=0.05, tol=.Machine$double.eps^.8) 

     Two-sample comparison of proportions power calculation 

              n = 2571429
             p1 = 9e-05
             p2 = 0.0001150142
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

You can see that p2 changed. If we insert the updated value for p2 and solve for n we get

power.prop.test(
  p1=original_conversion_rate, 
  p2=0.0001150142,
            power=0.8, 
            sig.level=0.05)

     Two-sample comparison of proportions power calculation 

              n = 2571429
             p1 = 9e-05
             p2 = 0.0001150142
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

and you get the same n.

power.prop.test function not interchangeable

1 Answers1