Why using a custom proximity function in proxy::dist makes it so slow in R?

Question

I am trying to use proxy::dist function with a custom distance matrix but what I have now is very slow.

This is a reproducible example of how I call my custom function:

set.seed(1)
test <- matrix(runif(4200), 60, 70)
train <- matrix(runif(4200), 60, 70)
dMatrix <- proxy::dist(x = test, y = train, method = customDTW,
                     by_rows = T, 
                     auto_convert_data_frames = T)

which is supposed to calculate the distance between each time series in test matrix with all time series in the train matrix (each row being a time series).

My custom function is:

customDTW <- function(ts1, ts2){

  d <- dtw(ts1, ts2,
      dist.method = "Euclidean",
      window.type = "sakoechiba",
      window.size = 20
  )
  return(d$distance)
}

The problem is that, comparing to when I use method="DTW", or even to the case where I calculate the distance matrix by myself, this is extremely slower, and as the length of the time series or the number of them grows, it get slower exponentially. Of course this is rooted in the nested loop, but I am surprised by the scale of the effect. There must be another reason that I am not seeing it.

My question is that how else I could implement my customDTW to make it faster, using proxy::dist?

This is my little experiment on the execution time:

Execution time for 60X7 (using proxy::dist + customDTW)

user  system elapsed 
2.852   0.012   2.867

Execution time for 60X70 (using proxy::dist + customDTW)

user  system elapsed 
5.384   0.000   5.382

Execution time for 60X700 (using proxy::dist + customDTW)

user  system elapsed 
509.088  18.652 529.115

Execution time for 60X700 (without using proxy::dist)

user  system elapsed 
26.696   0.004  26.753

score 2 · Answer 1 · answered May 18 '18 at 18:04

2

DTW is slow by nature Have you considered trying to use dtwclust (parallelized implementation of dtw)

https://github.com/asardaes/dtwclust

https://cran.r-project.org/web/packages/dtwclust/vignettes/dtwclust.pdf

answered May 18 '18 at 18:04

Carlos Santillan

1,077
7
8

I am going to need to compare the time factor of different variations of DTWs with one another. Since not all of the implementations are publicly available, I need to stick to my custom functions. This is why I am trying to avoid `dtwclust` package. – Azim May 18 '18 at 18:50
1

Your "own custom functions" are still using other's code (from the dtw package), so I don't see why you couldn't use the versions from the dtwclust package. [This](https://cran.r-project.org/web/packages/dtwclust/vignettes/timing-experiments.html#dtw-1) might be interesting if you care about timings. – Alexis May 19 '18 at 20:58

score 1 · Answer 2 · answered May 18 '18 at 22:14

1

This is what I found that seems to improve the speed, but it is still not as fast as I expect it to be. (Any other idea is still very welcome.)

The trick is to register the custom distance function with proxy (i.e., Registry of proximities here) so that you can use it like a built-in distance measure. So, first:

proxy::pr_DB$set_entry(FUN = customDTW, names=c("customDTW"),
                         loop = TRUE, type = "metric", distance = TRUE)

and now you can use it as if it was already in the proxy package.

dMatrix <- proxy::dist(x = test, y = train, method = "customDTW",
                         by_rows = T,
                         auto_convert_data_frames = T)

Note: If you want to use this method, then the customDTW method has to deal with one pair of time series, instead of all of them. So the customDTW would look like this:

customDTW2 <- function(ts1, ts2){

  d <- dtw(ts1, ts2,
      dist.method = "Euclidean",
      window.type = "sakoechiba",
      window.size = 20
  )
  return(d$distance)
}

For more, see ?pr_DB.

answered May 18 '18 at 22:14

Azim

1,596
18
34

Any other ideas? – Azim May 18 '18 at 22:14
You can look at the function the dtw package itself registers: defined [here](https://github.com/cran/dtw/blob/master/R/dtw-internal.R#L22) and registered [here](https://github.com/cran/dtw/blob/master/R/zzz.R#L18). You'll notice it uses the `distance.only=TRUE` parameter. – Alexis May 19 '18 at 20:56
I am not sure if I understand your suggestion. Does this help to use a custom function in a faster fashion? – Azim May 19 '18 at 23:58
1

I'm just saying that the dtw package already registers a function with proxy, and this proxy version is simply calling the `dtw::dtw` function, but it always sets `distance.only=TRUE` to speed up its calculations. Your `customDTW` function is essentially the same, just with pre-specified parameter values. However, you should also set `distance.only` to `TRUE` if you want to mirror what the dtw's proxy version does. – Alexis May 20 '18 at 00:58
Yes, but that was just a sample function. The idea is to be able to change it with any other custom function. – Azim May 20 '18 at 01:31
Check my other answer for that. – Alexis May 20 '18 at 09:47

Alexis · Accepted Answer · 2018-05-19T21:12:25.277

1

R is an interpreted language, and under the hood it is implemented in C. The proxy package is, as far as I understand, using R's interpretation capabilities from within C to call R code several times, but that still can't avoid the interpretation's overhead, so almost any "pure" R implementation will be slower.

Specifying loop=TRUE when registering a function with proxy means that the aforementioned will happen (proxy will interpret the R code several times to fill the distance matrix). If you really want to speed things up, you'd need to implement the filling itself in C/C++, and register the function with proxy with loop=FALSE; this is what dtwclust does (among other things).

You might want to look at the parallelDist package if you want to test your own custom C/C++ functions, even if you don't want to use parallelization.

edited May 19 '18 at 21:12

answered May 19 '18 at 21:06

Alexis

4,950
1
18
37

Thanks for the answer. But keeping the interpretation's time aside, in my experiments, I kept the number of rows the same (i.e., 60) as I increased the number of columns (i.e., 7, 70, 700). This was going to keep calling (interpreting) the custom function 60 times in all four experiments. So, if the bottleneck was the interpretation, it should have been almost equally slow in all four experiments. But as you see, it gets worst as the length of the time series grows. I am looking for a solution/explanation for this behavior. – Azim May 20 '18 at 14:27
Your assumptions are a bit off. For a matrix with 60 rows, proxy will create a 60x60 cross-distance matrix, which means 3600 DTW calculations. Furthermore, DTW's time complexity increases quadratically with the series' length (at least in simplified theory), so *each* of those 3600 calculations' time increases by a factor of 100 if all series' lengths increase by a factor of 10. – Alexis May 20 '18 at 14:53
I guess I just need to know the effect of the interpretation, in your last comment. I understand the effect that the length of time series have on the total execution time, but not the role of interpretation since we are calling the same function, the same number of times in all experiments. – Azim May 20 '18 at 15:23
I'm still not sure your expectations are realistic. Just as an example, the other day I computed a 10,000x10,000 DTW distance matrix with parallelization (*24* threads) and it took around 40 minutes. No R interpretation came into play. – Alexis May 20 '18 at 18:06

Why using a custom proximity function in proxy::dist makes it so slow in R?

3 Answers3