Fast numpy indexing for interpolation of an image

Question

I have an image that has a bunch of dead pixels in it. In python, I have one numpy array that will hold the final image, and I have another boolean numpy array of the same shape that indicates which pixels need to be filled in.

I want to fill in the dead pixels by taking the average of the 8 surrounding pixels, but only if they actually hold data. For example, if I have this (N means there is no data there, ? is the pixel to fill in):

1 2 N
N ? 2
N N 5

I will fill in the ? with (1+2+2+5)/4 .

Right now, I do this with a for loop as follows. outFrame holds the final image, while populated is the boolean array indicating which pixels have been filled in:

        # Loop through each pixel in the image
        for row in range(height):
            for col in range(width):
                # Check to see if the pixel needs to be filled in
                if not populated[row,col]:
                    # Check for boundary conditions
                    xmin = max(0,col-1)
                    xmax = min(width-1,col+1)
                    ymin = max(0,row-1)
                    ymax = min(height-1,row+1)

                    # Find the 8 surrounding values
                    vals = outFrame[ymin:ymax+1,xmin:xmax+1]
                    mask = populated[ymin:ymax+1,xmin:xmax+1]

                    # Find the average of only the populated pixels
                    if vals[mask].size != 0:
                        outFrame[row,col] = np.mean(vals[mask])

Obviously, python looping is slow but I can't figure out any numpy indexing magic to get this behavior. Is there a way to do this functionality quickly in python?

EDIT: I tried using the opencv inpainting function as follows:

mask = (1*~populated).astype(np.uint8)
outFrame = cv2.inpaint(outFrame,mask,3,cv2.INPAINT_NS) # This is very slow

However, this was 10x slower than my original method (my method takes ~3-6 seconds, while the inpainting method took 60 seconds). I have a very large amount of dead pixels, and I think this is the reason it is slow for this method

EDIT: Here is an example image.
Raw image: https://i.stack.imgur.com/2QXc1.jpg
After interpolation: https://i.stack.imgur.com/8BHZn.jpg

Simply do a 2D convolution with 3x3 kernel of the image and the mask and divide them up? — Divakar, Jan 30 '20 at 21:24
OpenCV will do that extremely quickly - it's called *"inpainting"*, see here... https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_photo/py_inpainting/py_inpainting.html — Mark Setchell, Jan 30 '20 at 22:30
I edited to show the results using inpainting. It was extremely slow for me — Jay Jackman, Jan 30 '20 at 22:44
@Divakar, I thought about doing that, but I'm not sure how to ignore the dead pixels if i use, e.g., cv2.convolve2D appropriately — Jay Jackman, Jan 30 '20 at 22:47
@JayJackman The dead pixels are accounted for when you do the convolution for the mask. Without dead pixels, a direct convolution of the image would suffice. — Divakar, Jan 31 '20 at 05:20
You do 2 `min()` and 2 `max()` checks, i.e. at least 4 `if` statements for every single pixel to test if you are at the edge. If you start your loop on the second pixel and end on the second-to-last pixel in each row and column, you can avoid that. — Mark Setchell, Jan 31 '20 at 08:03
There's a `generic_filter()` in SciPy that you could maybe try... https://stackoverflow.com/a/54440152/2836621 — Mark Setchell, Jan 31 '20 at 08:20
You could also share some sample images so folk can assist you better without having to guess the size of your image and the type/frequency of defects you are trying to correct.. — Mark Setchell, Jan 31 '20 at 08:21
I just wanted to thank you for the really neat test image- that's one I hadn't thought of (designing around) and it certainly would work things hard. — J.Hirsch, Dec 18 '20 at 14:39

score 2 · Answer 1 · answered Feb 04 '20 at 09:06

It's rather hard to help you because you haven't provided a Minimum Complete Verifiable Example of your code with all the import statements and code showing how you open your images, or even indicated whether you are using OpenCV or PIL or skimage. Also, you haven't provided the second file with the mask of all the points that need in-painting, nor do I know what you are actually trying to achieve, so for the moment, I am just trying to provide a method that looks to me like it gets a similar result to the one you show.

Anyway, here's a method that uses morphology and takes 100 microseconds on my Mac - which may not be anything like whatever you are using ;-)

#!/usr/bin/env python3

import cv2
import numpy as np

# Load image and make into Numpy array
im = cv2.imread('holey.png')

# Use morphology to find maximum pixel in each 3x3 square
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3,3))
morph = cv2.morphologyEx(im, cv2.MORPH_DILATE, kernel)

# Save result
cv2.imwrite('result.png', morph)

I think this should be pretty good! I didn't think of morphological operations because I thought they just worked with binary images (i know my image looks binary but it wont be in general). It's very fast for me (0.01 seconds) which is great. Thank you! — Jay Jackman, Feb 05 '20 at 00:09

score 1 · Accepted Answer · edited Jan 04 '21 at 16:58

I found the perfect method for my use case, thought I would post it. I believe it has the exact functionality of my original for loop method, while performing ~300 times faster! (3-4 seconds to 0.01 seconds)

import cv2
image = <READ IMAGE IN>
populated = <BOOLEAN MASK OF SAME SHAPE AS image>

image[~populated] = 0
blurred = cv2.boxFilter(image,-1,(3,3),normalize=False)
countMask = cv2.boxFilter(populated.astype('int'),-1,(3,3),normalize=False)
image[countMask != 0] = blurred[countMask != 0] / countMask[countMask != 0]

The first line sets the not populated pixels to 0, so they don't increase the sum for the blur step.
The second line blurs the image without normalization, which, for every pixel, sums the pixels around it.
The third line "blurs" the mask, which results in each pixel counting the number of valid pixels around it.
Lastly, the fourth line fills in the not populated pixels by normalizing the blurred pixels with the valid number

Credit to Divakar, he had the right idea.

Fast numpy indexing for interpolation of an image

2 Answers2