3

I wrote a threshold function TH(arr, threshold) that takes in a 2D array of vectors [u,v], and sets u and v to 0 if they both have an absolute value lower than a specified threshold.

The function consists of 2 for-loops and does the job, but is compute-intensive (I am running it on large datasets).

Examples:

[u, v] --> Output (threshold = 1)

[2, 2] --> [2, 2]

[2, .1] --> [2, .1]

[.1,.1] --> [0, 0]

What other methods/functions can I use to solve this problem more efficiently (Using list comprehension or other methods)?

Here's some code: 

import numpy as np
import time
start = time.time()

def TH(arr, threshold):
    for idx, value in enumerate(arr):
        for i, item in enumerate(value):
            if np.abs(item[0]) < threshold and np.abs(item[1]) < threshold:
                arr[idx][i][0] = 0.0
                arr[idx][i][1] = 0.0
    return arr

a = np.array([[[.5,.8], [3,4], [3,.1]],
              [[0,2], [.5,.5], [.3,3]],
              [[.4,.4], [.1,.1], [.5,5]]])

a = TH(a, threshold = 1)
print(a)

end = time.time()
print("Run time: ", end-start)

Output: 

[[[0.  0. ]
  [3.  4. ]
  [3.  0.1]]

 [[0.  2. ]
  [0.  0. ]
  [0.3 3. ]]

 [[0.  0. ]
  [0.  0. ]
  [0.5 5. ]]]

Run time:  0.0009984970092773438
Honey Gourami
  • 150
  • 11

1 Answers1

2

Simply slice with the two elements along the last axis and perform the same operations in a vectorized manner to get a mask and finally index with the mask into the input array to assign 0s -

mask = (np.abs(arr[...,0]) < threshold) & (np.abs(arr[...,1]) < threshold)
arr[mask] = 0

Note that arr[...,0] is another way to put arr[:,:,0] and is meant to slice a generic ndarray along the last axis. Similarly, for arr[...,1].

Alternatively, pre-compute the absolute values and use them to compare against threshold and look for all matches against last axis to get the same mask -

ab = np.abs(arr)
mask = (ab < threshold).all(-1)

Or, use the same slicing method after computing absolute values -

mask = (ab[...,0] < threshold) & (ab[...,1] < threshold)

For large arrays, we can also leverage numexpr module -

import numexpr as ne

m0 = ne.evaluate('abs(arr)<threshold')
mask = m0[...,0] & m0[...,1]

Timings -

In [209]: arr = np.random.rand(1080,1920,2)

In [210]: threshold = 1

In [211]: %timeit (np.abs(arr[...,0])<threshold) & (np.abs(arr[...,1])<threshold)
100 loops, best of 3: 10.2 ms per loop

In [212]: %timeit np.abs(arr).all(1)
10 loops, best of 3: 34.5 ms per loop

In [213]: %%timeit
     ...: ab = np.abs(arr)
     ...: (ab[...,0] < threshold) & (ab[...,1] < threshold)
     ...: 
100 loops, best of 3: 11 ms per loop

In [214]: %%timeit
     ...: m0 = ne.evaluate('abs(arr)<threshold')
     ...: m0[...,0] & m0[...,1]
     ...: 
100 loops, best of 3: 4.79 ms per loop
Divakar
  • 218,885
  • 19
  • 262
  • 358
  • I updated 2 of my threshold functions so far. Here are the execution times of the different approaches (Running on 25 frames of a video): My approach, 322.09 sec. First approach you proposed, 28.78 sec. Second approach, 29.32 sec. Third approach, 28.36 sec. On average, all of your proposed solutions seem to be 10x faster than my initial function. Thank you for your help and for the clear explanation! – Honey Gourami May 19 '19 at 16:22
  • 1
    @MarkH Appreciate you coming back with those perf figures! Curious - What's the typical shape of `arr` that you are working with? – Divakar May 19 '19 at 16:24
  • Each array that I am using holds data representing some feature of a video frame (RGB values, Optical Flow vectors [u,v], Magnitudes, Angles...). So my arrays have shapes similar to the size of the frames (Ex: (1080,1920,2)). After threshold & cleanup, I am storing the good values in an array to train a machine learning model on. This array has the shape (51 840 000, 5) for 25 frames. Context: I am working on Human Action Recognition. – Honey Gourami May 19 '19 at 17:08
  • 1
    @MarkH Added one more with `numexpr` that makes use of multi-cores. That could give further boost to perf. – Divakar May 19 '19 at 17:22
  • I tested it and did not see any difference in timing. Could the improvement in timing per loop be so small that the overall change is negligible? (10.2-4.79)ms * 25 loops = 135.25ms faster when running on 25 frames. Note: There is some variance of about +-0.5s in my code due to the mathematical models that I am using, which is probably why I won't see a difference. – Honey Gourami May 20 '19 at 01:30
  • @MarkH Probably because you are timing the entire code and not just this portion of code. – Divakar May 20 '19 at 04:37