0

I have a large array in R, let's say,

A = array(rnorm(100*100*30*30*100),dim=c(100,100,30,30,100))

I would like to find the first entry in the 4th dimension that is less than 1:

first_entry = apply(A,c(1,2,3,5),function(x) min(which(x < 1)))

Because the array is so large, this takes a while, around 20 seconds on my computer, and I'm going to do this for lots of arrays in the future. A for loop is about the same speed. I've also tried using parApply, but it takes around the same time, if not longer. Probably my function is not "complex" enough for parallelization to realize a speed gain. Is there a faster way to do this? And actually, what I would like to be able to do ideally is set the values in another array B (that has the same dimensions as A) to 0. So, something like,

B[first_entry] = 0

Note that this doesn't work given the current output of "apply" (above), since the dimension of first_entry there is 100x100x30x100.

Alex
  • 77
  • 1
  • 6
  • 1
    It is better to show a small data for testing purpose. Instead of the `min`, you can subset the first element with index i.e. `which(x < 1)[1]`. Also, by looking at the code, do you need the minimum value of those that are less than 1 or the first index where it is less than one? – akrun Nov 09 '18 at 19:15
  • @akrun Ooh yes, duh, that helps. :) I need the first index, though as I mentioned, it'd be nice if I could output the result in a way that I could call on the corresponding values in another array, B, that's the same size as A. – Alex Nov 09 '18 at 19:18

1 Answers1

0

I simplified your example a little bit (my machine may be slower than yours!), and added some time measurement. When I use parApplym rather then applym the code is a little faster (2.4 seconds with parApply vs 3.8 with apply).

If parApply indeed produces similar results on your machine, have you consider running the wider tasks in a multi-threaded fashion instead? You mentioned you will be doing this for a lots of arrays. So rather then making the look up process faster, could you run multiple look ups simultaneously? 

# Create data
A = array(rnorm(100*100*30),dim=c(100,100,30))

# Using apply
start.time <- Sys.time()
first_entry = apply(A,c(1,2,3),function(x) min(which(x < 1)))
print(Sys.time() - start.time)

# Load packages
library(parallel)

# Create cluster and export
nrCores <- detectCores()
cl <- makeCluster(nrCores)
clusterExport(cl=cl, varlist=c("A"))

# Using parApply
start.time <- Sys.time()
first_entry = parApply(cl=cl,A,c(1,2,3),function(x) min(which(x < 1)))
print(Sys.time() - start.time)
MatAff
  • 1,250
  • 14
  • 26