1

I am trying to calculate the entropy of 3D patches with sliding windows from a larger 3D array. I can't seem to find a way of optimising the code to run with any reasonable speed.

My current working approach uses nested for loops taking each coord of the larger array and calculating the entropy of the patch with that coord as the starting point.

I'd really like to be able to run this operation in parallel but can't seem to get pool.apply() working, is it possible to run this in parallel?

Here is my working code:

def get_entropy_of_block(data):
    value,counts = np.unique(data, return_counts=True)
    entropy_of_block = entropy(value, counts)
    if np.isnan(entropy_of_block):
        entropy_of_block = 0
    return entropy_of_block


def output_entropy_versions(mask, window = 5):    
        mask = np.pad(mask, (0,window - 2), 'edge')
        blocks = view_as_windows(mask, (window,window,window),step=1)
        entropy_mask = np.zeros(shape=(blocks.shape[0], blocks.shape[1], blocks.shape[2]))
        for x in range(blocks.shape[0]):
            print(x)
            for y in range(blocks.shape[1]):
                for z in range(blocks.shape[2]):
                    entropy_mask[x,y,z] = get_entropy_of_block(blocks[x,y,z,:,:])
        return entropy_mask

And here is the parallel attempt

def output_entropy_versions_parallel(mask, window = 5):    

        mask = np.pad(mask, (0,window - 2), 'edge')
        blocks = view_as_windows(mask, (window,window,window),step=1)
        entropy_mask = np.zeros(shape=(blocks.shape[0], blocks.shape[1], blocks.shape[2]))
        for x in range(blocks.shape[0]):
            print(x)
            for y in range(blocks.shape[1]):
                    res = [pool.apply(get_entropy_of_block, args = (blocks[x,y,z,:,:])) for z in range(blocks.shape[2])]
                    entropy_mask[x,y,:] = res
        return entropy_mask

Running this I get the following:


<ipython-input-10-8c3d4ca9d313> in output_entropy_versions(mask, window)
     24                 print(x)
     25                 for y in range(blocks.shape[1]):
---> 26                         res = [pool.apply(get_entropy_of_block, args = (blocks[x,y,z,:,:])) for z in range(blocks.shape[2])]
     27                         entropy_mask[x,y,:] = res
     28             return entropy_mask

> <ipython-input-10-8c3d4ca9d313> in get_entropy_of_block(data)
     10 def get_entropy_of_block(data):
     11         value,counts = np.unique(data, return_counts=True)
---> 12         entropy_of_block = entropy(value, counts)
     13         if np.isnan(entropy_of_block):
     14             entropy_of_block = 0

E:\Anaconda\lib\site-packages\scipy\stats\_distn_infrastructure.py in entropy(pk, qk, base)
   2505     """
   2506     pk = asarray(pk)
-> 2507     pk = 1.0*pk / np.sum(pk, axis=0)
   2508     if qk is None:
   2509         vec = entr(pk)

TypeError: unsupported operand type(s) for *: 'float' and 'generator'

  • can you give some context from the stacktrace? seems like a `generator` needs to be made into a `numpy.ndarray` at some point – Nathan McCoy Jul 11 '19 at 15:37
  • `E:\Anaconda\lib\site-packages\scipy\stats\_distn_infrastructure.py in entropy(pk, qk, base) 2505 """ 2506 pk = asarray(pk) -> 2507 pk = 1.0*pk / np.sum(pk, axis=0) 2508 if qk is None: 2509 vec = entr(pk) TypeError: unsupported operand type(s) for *: 'float' and 'generator'` – Chris Culley Jul 11 '19 at 15:59
  • looks like `pool.apply` is returning a `generator`? – Nathan McCoy Jul 11 '19 at 16:09
  • I've just realised, I have copied the wrong 'working code' *(doh)*, "for z in range(blocks.shape[2]): entropy_mask[x,y,z] = get_entropy_of_block(blocks[x,y,z,:,:])" - replaces the pool.apply – Chris Culley Jul 11 '19 at 16:09
  • 1
    please update your code to reflect and put the stack trace for the relevant code in the question – Nathan McCoy Jul 11 '19 at 16:14
  • Ok - all done, thanks (sorry, newb) – Chris Culley Jul 11 '19 at 16:23
  • `value` is a generator, check your data types – Nathan McCoy Jul 11 '19 at 16:31

0 Answers0