How to make this loop parallel and faster?

Question

I have this set of images from which I want to create a set of sub images with a stride with sub image size of 128*128, original image must be greater than this size (row and column both), I have created the following functions :

def sliding_window(image, stride, imgSize):
height, width, _ = image.shape
img = []
a1 = list(range(0, height-imgSize+stride, stride))
a2 = list(range(0, width-imgSize+stride, stride))
if (a1[-1]+imgSize != height):
    a1[-1] = height-imgSize
if (a2[-1]+imgSize != width):
    a2[-1] = width-imgSize
for y in a1:
    for x in a2:
        im1 = image[y:y+imgSize, x:x+imgSize, :]
        img.append(np.array(im1))
return img

and the main code snippet from where I call this definition :

im_counter = 0

image_data = []
image_label = []
for cl in file_images:
    for img_file in data[cl]:
        path = img_path + cl + "/" + img_file
        im = image.load_img(path)
        im = image.img_to_array(im)
        im_counter += 1
        if(im_counter % 500 == 0):
            print("{} images processed...".format(im_counter))
        if (im.shape[0] >= SIZE and im.shape[1] >= SIZE):
            img = sliding_window(im, STRIDE, SIZE)
            for i in range(len(img)):
                if(img[i].shape[2] >=3):
                    temp_img = img[i]
                    temp_img = preprocess_input(temp_img)
                    image_data.append(temp_img)
                    del temp_img
                    gc.collect()
                    image.append(class_dictionary[cl])

Now, the above code snippet takes forever to run on only 3000 images (takes at least 25 hours with utilizing only 1 CPU core), I want to make this faster, I have server access, the CPU has many cores, so can you please suggest a parallelized version of it so that it runs faster ?

NOTE : The sequence of subimages in which it is returned from the original image matters very much, No arbitrary sequence of image is allowed.

It looks like you're adding images to `image_data` in a very specific order. You also have a variable `image` and a variable `class_dictionary` that are being accessed at the end without any clear indication of what they are doing. Also, how big are these images? — Frank Yellin, Nov 07 '21 at 06:27
@FrankYellin The codes are : `file_images = ['negative', 'positive']` and `class_dictionary = {'negative': 0, 'positive': 1}` and the image variable is from : `from keras.preprocessing import image` and the original image sizes vary from `min(width): 128 ,max(width): 5183 ,min(height): 128 ,max(height): 5202`. — Aanzil Akram, Nov 07 '21 at 06:41

score 1 · Accepted Answer · answered Nov 07 '21 at 06:43

Here is a rough outline of something you can try.

def main():
    # Create a list of tuples consisting of the file path, and the class
    # dictionary info for each of the cl arguments
    args = []
    for cl in file_images:
        for img_file in data[cl]:
            path = img_path + cl + "/" + img_file
            args.append((path, class_dictionary[cl]))

    with multiprocessing.Pool(processes=30) as pool:   # or however many processes
        image_counter = 0
        # Use multiprocessing to call handle_on_image(pathname, info)
        # and return the results in order
        for images, info in pool.starmap(handle_one_image, args):
            # Images is a list of returned images.  info is the class_dictionary info that we passed
            for image in images:
                image_counter += 1
                image_data.append(image)
                image_label.append(info)

def handle_one_image(path, info):
    image_data = []
    im = image.load_img(path)
    im = image.img_to_array(im)
    if (im.shape[0] >= SIZE and im.shape[1] >= SIZE):
        img = sliding_window(im, STRIDE, SIZE)
        for i in range(len(img)):
            if(img[i].shape[2] >=3):
                temp_img = img[i]
                temp_img = preprocess_input(temp_img)
                image_data.append(temp_img)
        return image_data, info
    else:
        # indicate that no images are available
        return [], info

Umm. Err. Never mind. Are there just two images? Then this won't work at all. I assumed that `file_images` was a large set of file images, and that makes it easy to parallelize. — Frank Yellin, Nov 07 '21 at 06:45
No, actually `data[]` is a dictionary containing the file names of images, so I had to use a for loop to access those, so ultimately the `path` variable contains the address of all images from each iteration of the for loop. can you try and modify your code and help me out plz. ? — Aanzil Akram, Nov 07 '21 at 06:52
Thanks, that code works like a charm, I have already generated the sub images, from over a day to in just 3 minutes. Just needed some minor changes from my side. — Aanzil Akram, Nov 07 '21 at 07:15
Glad to help. I figured you'd need to make some modifications, but at least this would give you an idea of how to get started. You can, of course, set processes=30 to whatever you want. Or even leave it off and Python will figure out how many cores your machine has. I purposely left out the call to `gc.collection()` because I didn't think that was helping. — Frank Yellin, Nov 08 '21 at 00:58

How to make this loop parallel and faster?

1 Answers1