1

I've had following codes that use Python and OpenCV. Briefly, I have a stack of image taken at different focal depth. The codes pick out pixels at every (x,y) position that has the largest Laplacian of Guassian response among all focal depth(z), thus creating a focus-stacked image. Function get_fmap creates a 2d array where each pixel will contains the number of the focal plane having the largest log response. In the following codes, lines that are commented out are my current VIPS implementation. They don't look compatible within the function definition because it's only partial solution.

# from gi.repository import Vips

def get_log_kernel(siz, std):
    x = y = np.linspace(-siz, siz, 2*siz+1)
    x, y = np.meshgrid(x, y)
    arg = -(x**2 + y**2) / (2*std**2)
    h = np.exp(arg)
    h[h < sys.float_info.epsilon * h.max()] = 0
    h = h/h.sum() if h.sum() != 0 else h
    h1 = h*(x**2 + y**2 - 2*std**2) / (std**4)
    return h1 - h1.mean()

def get_fmap(img):    # img is a 3-d numpy array.
    log_response = np.zeros_like(img[:, :, 0], dtype='single')
    fmap = np.zeros_like(img[:, :, 0], dtype='uint8')
    log_kernel = get_log_kernel(11, 2)
    # kernel = get_log_kernel(11, 2)
    # kernel = [list(row) for row in kernel]
    # kernel = Vips.Image.new_from_array(kernel)
    # img = Vips.new_from_file("testimg.tif")
    for ii in range(img.shape[2]):           
        # img_filtered = img.conv(kernel)
        img_filtered = cv2.filter2D(img[:, :, ii].astype('single'), -1, log_kernel)
        index = img_filtered > log_response
        log_response[index] = img_filtered[index]
        fmap[index] = ii
    return fmap

and then fmap will be used to pick out pixels from different focal planes to create a focus-stacked image

This is done on an extremely large image, and I feel VIPS might do a better job than OpenCV on this. However, the official documentation provides rather scant information on its Python binding. From the information I can find on the internet, I'm only able to make image convolution work ( which, in my case, is an order of magnitude faster than OpenCV.). I'm wondering how to implement this in VIPS, especially these lines?

log_response = np.zeros_like(img[:, :, 0], dtype = 'single')

index = img_filtered > log_response

log_response[index] = im_filtered[index]

fmap[index] = ii
user3667217
  • 2,172
  • 2
  • 17
  • 29

2 Answers2

1

log_response and fmap are initialized as 3D arrays in the question code, whereas the question text states that the output, fmap is a 2D array. So, I am assuming that log_response and fmap are to be initialized as 2D arrays with their shapes same as each image. Thus, the edits would be -

log_response = np.zeros_like(img[:,:,0], dtype='single')
fmap = np.zeros_like(img[:,:,0], dtype='uint8')

Now, back to the theme of the question, you are performing 2D filtering on each image one-by-one and getting the maximum index of filtered output across all stacked images. In case, you didn't know as per the documentation of cv2.filter2D, it could also be used on a multi-dimensional array giving us a multi-dimensional array as output. Then, getting the maximum index across all images is as simple as .argmax(2). Thus, the implementation must be extremely efficient and would be simply -

fmap = cv2.filter2D(img,-1,log_kernel).argmax(2)
Divakar
  • 218,885
  • 19
  • 262
  • 358
  • Thanks. I made a mistake when providing this minimal example. I've edited my post. It's really good to know that `cv2.filter2d` has such usage. I'll certainly give it a try. However, VIPS is by default using multiple cores, while with OpenCV, I'll have to implement this myself. And adding multiprocessing to OpenCV will also add some overhead. I don't think the `multiprocessing` module plus OpenCV will perform close to VIPS. I tested Gaussian filtering on a 3000x4000 imagewith a 11x11 kernel. VIPS finished the task in 0.0025 second. OpenCV took 0.1334 seconds. – user3667217 Oct 18 '15 at 15:08
  • @user3667217 So, you have already implemented the solution in VIPS? Si so, share as an answer here? Well, I haven't heard about it until today from you, so thanks! Let me know any other thoughts on this! That VIPS, I would certainly looks interesting. – Divakar Oct 18 '15 at 16:26
  • I haven't got a full solution in VIPS, that's why I ask about how to perform logical operation and indexing in VIPS. Apparently VIPS is actively developed ( currently version 8 already) but much less known. What I managed to do is nothing too far from existing python manual. I only got it to perform Gaussian filtering and using my own kernel for filtering. I'm still exploring it. I'll the codes in my OP. VIPS is truly amazing. For example, it spent 0.0011 seconds reading image, while openCV spent 0.043 seconds. It's just mindbogglingly fast. Definitely worth trying ! – user3667217 Oct 18 '15 at 17:35
  • @user3667217 Crazy stuffs you are sharing with me! I mean I have been using OpenCV for quite a long time. So, from what you are saying this VIPS seems like a new thing and definitely exciting! I will surely give it a go sometime. But on this question, I don't think I can help with VIPS. Would be worth waiting to see someone has more insights on it. – Divakar Oct 18 '15 at 17:38
  • Have you ever tried OpenCV with TBB ? My above results are from single threaded OpenCV. OpenCV + TBB is one alternative but I can't find many examples . I still don't think they together out-perform VIPS, thuogh. But of course, VIPS is way more limited in terms of number of functions it performs. – user3667217 Oct 18 '15 at 18:07
  • @user3667217 No, so far I have not worked with any parallelization on OpenCV, but definitely on the cards very soon in near future. I think multiprocessing is one thing I would start off with. Parallelization is one area I really need to explore on Python! So far, all of my "parallelization" has been limited to NumPy. – Divakar Oct 18 '15 at 18:09
  • The big thing vips does is stream images. It can load, process and save all in parallel, streaming the image though your set of CPU cores. This gives an enormous speedup and memory saving for large images. Your gaussblur test may be unfair: vips will use a separable convolution there, and of course that's much faster than a full 2D convolution. – jcupitt Oct 19 '15 at 08:10
  • @user894763 Very useful info there, thanks! Definitely on the cards for trying out after inspirations from you and OP! – Divakar Oct 19 '15 at 09:21
1

After consulting the Python VIPS manual and some trial-and-error, I've come up with my own answer. My numpy and OpenCV implementation in question can be translated into VIPS like this:

import pyvips

img = []
for ii in range(num_z_levels):
    img.append(pyvips.Image.new_from_file("testimg_z" + str(ii) + ".tif")

def get_fmap(img)
    log_kernel = get_log_kernel(11,2)  # get_log_kernel is my own function, which generates a 2-d numpy array.
    log_kernel = [list(row) for row in log_kernel]  # pyvips.Image.new_from_array takes 1-d list array.
    log_kernel = pyvips.Image.new_from_array(log_kernel)  # Turn the kernel into Vips array so it can be used by Vips.
    log_response = img[0].conv(log_kernel)

    for ii in range(len(img)):
        img_filtered = img[ii+1].conv(log_kernel)
        log_response = (img_filtered > log_response).ifthenelse(img_filtered, log_response)
        fmap = (img_filtered > log_response).ifthenelse(ii+1, 0)

Logical indexing is achieved through ifthenelse method :

result_img = (test_condition).ifthenelse(value_if_true, value_if_false)

The syntax is rather flexible. The test condition can be a comparison between two images of the same size or between an image and a value, e.g. img1 > img2 or img > 5. Like wise, value_if_true can be a single value or a Vips image.

jcupitt
  • 10,213
  • 2
  • 23
  • 39
user3667217
  • 2,172
  • 2
  • 17
  • 29
  • 1
    I'm the vips maintainer. Your code looks good. Did you see the vips `logmat` function? It might work for you. It creates a log mask http://www.vips.ecs.soton.ac.uk/supported/current/doc/html/libvips/libvips-create.html#vips-logmat use from Python as (for example) `Vips.Image.logmat(2, 0.1)`. It'll generate approximate integer masks and separable masks as well, which can give a useful speedup. You can use the string `"float"` instead of `Vips.Precision.FLOAT` if you wish. – jcupitt Oct 19 '15 at 08:04
  • 1
    (part 2) Do you need the `index` image? I'd think you could rewrite that as `fmap = (img_filtered > log_response).ifthenelse(ii + 1, 0)`, I'm probably missing something. I'd be curious how speed and memory use compares to opencv, if you have any benchmarks you could share. – jcupitt Oct 19 '15 at 08:04
  • Try: `x = Vips.Image.logmat(2, 0.1); x.matrixprint()` to see the log mat that vips makes. Add `precision = "float"` to get a float version. – jcupitt Oct 19 '15 at 08:27
  • Oh, you're the maintainer !! Thanks for your comment. I just began using VIPS last week and still have trouble finding where everything is. I couldn't quite understand the equation regarding the log kernel. Is it the same as mine? (I've just posted it in my question). I don't need `index` image, so what you suggested should be better. – user3667217 Oct 19 '15 at 08:27
  • There's a page in the manual with all the vips operations: http://www.vips.ecs.soton.ac.uk/supported/current/doc/html/libvips/func-list.html I usually ^F in that to find stuff, don't know if that helps. – jcupitt Oct 19 '15 at 08:29
  • Once I'm able to fully convert the parts covered by OpenCV to VIPS, I'd definitely love to provide benchmarks. One quick question here, though. I read that VIPS can only handle image of up to 2^31 pixels? It that correct? – user3667217 Oct 19 '15 at 08:30
  • 1
    Image axes are limited to 2^31, pixels to the square of that, so 2^62. I regularly process 200,000 x 200,000 pixel slide images on my laptop. It can do these huge images on 32-bit machines too, though that's less relevant these days, thank goodness. – jcupitt Oct 19 '15 at 08:34
  • The log kernel looks similar, but I'd check the one that vips is making (obviously). – jcupitt Oct 19 '15 at 08:36