1

It seems that for large N (say 2e6) the cut2 function in the Hmisc package throws an error

 y = cut2(rnorm(2000000,0,1),m=sqrt(2000000))

 Error in if (cj == upper) next : missing value where TRUE/FALSE needed
 In addition: Warning message:
 In (1:g) * nnm : NAs produced by integer overflow

I'm trying to obtain quantiles of my data, with m points in each quantile, and also record endpoints of each quantile. cut2 does this, but not very well for large N. Are there better alternatives?

stats134711
  • 624
  • 1
  • 5
  • 15

1 Answers1

1

Is this what you want?

cut3 = function(x, m) {
    p = seq(0, 1, by = m / length(x))
    q = quantile(x, probs = p, names = F)
    result = cut(x, breaks = q)
}

Testing it out:

x = rnorm(2e6)
m = sqrt(2e6)
qq = cut3(x, m)
summary(as.numeric(table(qq)))
# Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
# 1414    1414    1414    1414    1414    1415 
head(qq)
# [1] (0.4757,0.4779] (-1.021,-1.018] (0.4325,0.4344] (1.376,1.381]   (-2.156,-2.138] (0.1215,0.1233]
# 1414 Levels: (-4.964,-3.196] (-3.196,-2.981] (-2.981,-2.86] (-2.86,-2.766] (-2.766,-2.696] (-2.696,-2.637] ... (3.145,3.607]
Gregor Thomas
  • 136,190
  • 20
  • 167
  • 294
  • There are some slight differences, as I'm starting to see. `cut2` uses `m` as the minimum number of observations in each bin, while `cut3` forces each bin to have exactly the same number of observations. The latter results in some observations not being placed in bins. Is it possible to change this with `quantile` and `cut`? – stats134711 Sep 15 '16 at 02:30