0

Given a matrix, say m, is there any direct method to find top k values of m and then find exactly which column/row do they belong to. I couldn't find any on SO and hence, putting this question. My try on the above has been this:

set.seed(1729)
k=5 #top 5
m = matrix(round(runif(30),digits = 2),nr=10)
idx <- which(matrix(m %in% head(sort(m), k), nr = nrow(m)), arr.ind = TRUE)
print(m)
      [,1] [,2] [,3]
 [1,] 0.59 0.54 0.57
 [2,] 0.44 0.43 0.32
 [3,] 0.57 0.08 0.29
 [4,] 0.35 0.58 0.24
 [5,] 0.86 0.52 0.53
 [6,] 0.41 0.78 0.17
 [7,] 0.51 0.47 0.26
 [8,] 0.15 0.81 0.49
 [9,] 0.85 0.64 0.64
[10,] 1.00 0.78 0.95

print(idx)

        row col
[1,]   8   1
[2,]   3   2
[3,]   4   3
[4,]   6   3
[5,]   7   3

I am not sure if this is efficient because of the reason that I am sorting the entire values of a matrix rather than picking up those k values. I would like to assume k << length(m). Are there any efficient ways for a large matrix m, and also are there any methods which could help me with duplicates in the scenarios like when one wants to get top k column names

For example: with a matrix mm, I need to identify top 2 columns having least values. Here, for the following case I am expecting columns 1 and 2

mm = matrix(c(6,6,7,8,7,9,8,8,9), 3)
 print(mm)
     [,1] [,2] [,3]
[1,]    6    8    8
[2,]    6    7    8
[3,]    7    9    9
idx <- which(matrix(mm %in% head(sort(mm), 2), nr = nrow(mm)), arr.ind = TRUE)
print(idx)
      row col
[1,]   1   1
[2,]   2   1

But, here I get only one column, i.e.; 1 , In this case, output should be two different columns having least values viz. 1 and 2

Frank
  • 66,179
  • 8
  • 96
  • 180
rahulkmishra
  • 63
  • 6
  • 2
    For the first part see [this post](http://stackoverflow.com/questions/3692563/how-to-return-5-topmost-values-from-vector-in-r). Basically, you use `sort.int` and `partial = TRUE` to speed up your sort. – Barker Nov 18 '16 at 05:08
  • My apologies, I meant `partial = 1:k`. – Barker Nov 18 '16 at 05:18
  • Thanks for link! It is useful in some way. – rahulkmishra Nov 18 '16 at 10:14
  • For the new part, if you want to know which two columns contain the lowest values, maybe something like `which( data.table::frank(matrixStats::colMins(mm), ties.method="dense") %in% 1:2 )` – Frank Nov 18 '16 at 11:54

1 Answers1

1

Here's a comparison of the OP's approach, @Barker's suggestion to substitute in R's partial sorting functionality and a way using quantile:

# example data
set.seed(1729)
n = 1e6
k = 50
m = matrix(runif(n), nr=10)

# illustration of the quantile way
which(m <= quantile(m, k/length(m)), arr.ind = TRUE)
# or...
library(data.table)
setDT(melt(m))[ value <= quantile(value, k/.N) ]
#    Var1  Var2        value
# 1:    8  4945 1.471722e-06
# 2:    1  7025 1.856475e-05
# 3:    9  7480 4.518987e-05
# 4:   10  8378 1.877453e-05
# 5:    2  9043 3.262958e-05
# 6:    7  9925 1.327880e-05
# 7:    5 13571 5.097035e-05
# ...

# benchmark
microbenchmark::microbenchmark(times = 30,
  idx  = idx <- which(matrix(m %in% head(sort(m), k), nr = nrow(m)), arr.ind = TRUE),
  dtq  = dtq <- setDT(melt(m))[ value <= quantile(value, k/.N) ],
  idxp = idxp <- which(matrix(m %in% head(sort(m, partial = 1:k), k), nr = nrow(m)), arr.ind = TRUE),
  idxq = idxq <- which(m <= quantile(m, k/length(m)), arr.ind = TRUE)
)

# verifying, requires data.table 1.9.7+
fsetequal(as.data.table(idx), dtq[, .(row = Var1, col = Var2)])
fsetequal(as.data.table(idxp), dtq[, .(row = Var1, col = Var2)])
fsetequal(as.data.table(idxq), dtq[, .(row = Var1, col = Var2)])

which gives

Unit: milliseconds
 expr       min        lq      mean    median        uq       max neval  cld
  idx 145.01260 148.10571 155.27124 149.97761 152.45523 206.27179    30    d
  dtq  30.05910  33.11280  44.83088  35.02334  37.78721  90.92545    30  b  
 idxp 114.69501 118.23185 127.37992 119.50131 121.33241 175.41117    30   c 
 idxq  13.02406  14.47907  22.81266  16.41707  18.28308  68.53364    30 a

I took out the OP's rounding for this example. Tweaking the parameters n and k may lead to a different ranking of the approaches. My preferred way would be setDT(melt(m))[order(value, partial = 1:k)] but it looks like that's not available in R yet.

Frank
  • 66,179
  • 8
  • 96
  • 180
  • Thanks for benchmarking the results. I have started using idxq method. Any thoughts/insights on how to get unique top K columns, without duplicates? – rahulkmishra Nov 18 '16 at 06:38
  • @rahul I'm not sure what you mean by that. if it's significantly different, you might want to post it as a new question..? Otherwise, maybe you could edit this question to clarify, maybe by showing what it means for your example. – Frank Nov 18 '16 at 06:42
  • I have edited the question to provide explanation of what I mean by duplicates. Ties will be a better term. no? – rahulkmishra Nov 18 '16 at 10:12