2

I have a vector of 300 sentences, and I am trying to find elementwise JW distance using the stringdist package. The execution time for the naive implementation is too high, leading me to look for ways to reduce the runtime. I am trying to leverage the doParallel and foreach packages, but I'm not getting any significant speedup. This is how I am going about it.

library(foreach)
library(doParallel)
cl = makeCluster(detectCores())
registerDoParallel(cl)

sentence = # vector containing sentences 
jw_dist = foreach(i = 1:length(sentence)) %dopar% {
 temp = sentence[sentence!=sentence[i]]
 return(mean(1 - stringdist::stringdist(sentence[i],temp,method = "jw",nthread = 3))
  }
  stopCluster(cl)

I would really appreciate if someone can point out ways in which I can speed up this chunk of code.

WitchKingofAngmar
  • 212
  • 3
  • 17
  • So you want to calculate the pairwise distance between the single sentences? In your code you're using parallelism two times, first with `dopar` and then within the `stringdist` function where you specify the number of threads ... I don't think that's good practice – Val Jun 19 '17 at 12:21
  • Thank you for your suggestion. I have observed that the only speedup I get is when I use the `nthread` parameter inside the `stringdist` function. I tried running the code with the `nthread` parameter set to default and `dopar` and did not get any speedup. – WitchKingofAngmar Jun 19 '17 at 12:41
  • What's your reference time and machine setup? Also, I'm assuming correctly that you want to calculate the distance for each pair of sentences? – Val Jun 19 '17 at 12:54
  • I am planning to run this code with ~1000 sentences, for which this code runs in about 40 seconds. I want to bring it down in the neighborhood of 20 seconds. – WitchKingofAngmar Jun 19 '17 at 13:16
  • I'm running R version 3.2.2 on a 64 bit machine Running Ubuntu 17.04. And yes, I am looking to calculate the distance for each pair of sentences, – WitchKingofAngmar Jun 19 '17 at 13:23

1 Answers1

1

So it seems you're fighting with extreme overhead.

Instead of parallelizing on the single sentences, just split the task in some sizable chunks and let apply do the rest. I've chosen 10 chunks of 100 sentences each, possibly there's a faster combination but this one works much faster (at least for me) than what you asked for:

library(doParallel)
library(foreach)

# generate fake sentences

txt <- readLines(url('https://baconipsum.com/api/?type=all-meat&sentences=300&start-with-lorem=1&format=text'))

sentences <- strsplit(txt,'\\.\\s')[[1]]

sentences <- rep(sentences[sample(1:100,100)],10)

# pairwise combinations of sentences
cbn <- combn(1:length(sentences),2)

# simple timing
st <- Sys.time()

# Since you work on LINUX, you can use FORK
cl <-  makeCluster(detectCores(),type = 'FORK')
registerDoParallel(cl)


res <- foreach(ii = seq(1,1000,100),.combine = 'c') %dopar% {
  
  apply(cbn[,ii:(ii+99)],2,function(x) stringdist(sentences[x[1]],sentences[x[2]],method = "jw"))
   
}

stopCluster(cl)
Sys.time() - st

On my Ubuntu VM, this code runs in ~ 1.8 seconds.

Specs:

Ubuntu 64 bit
R version 3.4
8 CPU cores
32GB RAM Memory

HTH

Edit:

Maybe avoiding parallel-processing would be a good alternative in this case.

Using this lapply version, I can calculate the mean for each sentence in ~ 17 seconds:

res <- do.call(rbind,lapply(1:1000,function(ii) c(ii,1-mean(stringdist(sentences[ii],sentences[-ii],method = "jw")))))

This will give you a 2 column matrix with the index for each sentence and 1-mean of all distances to the respective sentence.

Community
  • 1
  • 1
Val
  • 6,585
  • 5
  • 22
  • 52
  • I am looking to calculate the mean JW distance for each entry in `sentence` with every other entry. As such, I will try storing all the pairwise distances from your method in a matrix or something and then calculate a row-wise mean. Thank you for the tip. – WitchKingofAngmar Jun 19 '17 at 14:20
  • I was thinking you need the pairwise comparisons. Considering the averages, I updated my solution with a `lapply` version that performs reasonably well. – Val Jun 19 '17 at 14:54
  • Using the method in the update, I can now work with 1000 sentences in just about 30 seconds. I think this is a good start. Thank you for your help ! – WitchKingofAngmar Jun 19 '17 at 16:03