0

professionals and students,

I have significance levels 10%,5% & 1% and I have computed the relative rejection frequency thanks to an answer on my previous question. 

replicate_sw10 = replicate(1000,shapiro.test(rnorm(10)))
table(replicate_sw10["p.value",]<0.10)/1000

> FALSE  TRUE 
> 0.909 0.091

But if I have done this for various sample sizes (T=10,30,50,100,500) and stored it manually via excel. Maybe there is an ever easier way to compute this in a function/list.

However how do I measure if it significantly different from significance levels? (The hint is the following: the rejection of a test can be modelled as a Bernoulli random variable)

Best regards

FinancialRiskManagerBE
  • 49
  • 8

1 Answers1

1

So, the easiest way to do this is.. so if you perform 1000 test, you would expect approximately 0.1 of your test to have a pvalue < 0.1. It's like a bernoulli trial like you said, and you can use a binomial test to see the probability of something as extreme as your result:

set.seed(100)
replicate_sw10 = replicate(1000,shapiro.test(rnorm(10)))
obs_significant = sum(replicate_sw10["p.value",]<0.1)

binom.test(obs_significant,n=1000,p=0.1)

    Exact binomial test

data:  obs_significant and 1000
number of successes = 118, number of trials = 1000, p-value = 0.06479
alternative hypothesis: true probability of success is not equal to 0.1
95 percent confidence interval:
 0.09865252 0.13962772
sample estimates:
probability of success 
                 0.118
StupidWolf
  • 45,075
  • 17
  • 40
  • 72
  • Can I thus conclude that it is always not equal to the significance level? because only the confidence interval is of additional information in comparison to my other post (your answer gives the same output except for the confidence interval) – FinancialRiskManagerBE Mar 15 '20 at 09:51
  • i don't quite get what you are trying to conclude.. All these simulation show that if you simulate data under the null hypothesis and repeat the test N times, you would expect say about 0.05 of N to be < 0.05 .. 0.1 to be < 0.1 and so on – StupidWolf Mar 15 '20 at 09:57
  • If the normality tests are well-specified, you expect to find rejections in 10%, 5%, 1% of the replications (confidence levels). This part is a test whether the sizes are significantly different from these significance levels – FinancialRiskManagerBE Mar 15 '20 at 10:16
  • 1
    Ok so I guess most likely not that different? – StupidWolf Mar 15 '20 at 10:21