Apply preprocessing to the dataset

Question

I am implementing a paper on image segmentation in pytorch. I am required to do some preprocessing steps but as I am trying it the first time so I am unable to incorporate them in the traditional pipeline. Following are the preprocessing steps-

1) N(w, h) = I(w, h) − G(w, h), (1) where N is the normalized image, I is the original image, and G is the Gaussian blurred image with kernel size 65*65 and 0 mean and standard deviation 10.

2)Normalizing the mean image and dividing each pixel by average standard deviation.

Following is my code snippet for the above steps-

def gaussian_blur(img):
    image = cv2.GaussianBlur(image,(65,65),10)
    new_image = img - image
return image

def normalise(img):
    img_normalised = np.empty(img.shape)
    img_std = np.std(img)
    img_mean = np.mean(img)
    img_normalized = (img-img_mean)/imgs_std

    for i in range(img.shape[1]):
        img_normalized[i] = (img_normalized - 
         np.mean(img_normalized))/np.std(img_normalized)
return img_normalized

I am really not sure how to add above functions in the traditional pytorch data-loaders pipeline like first I should load the dataset using ImageFolder and then apply or first apply and then use ImageFolder method.

I'm too lazy to provide a working code right now, but you can read about [Generic Transforms](https://pytorch.org/docs/stable/torchvision/transforms.html#generic-transforms) to do it in the easy way. — Berriel, Jul 09 '19 at 18:59
As @Berriel said, the Generic Transforms work fine, but, I'd also recommend you the [imgaug](https://imgaug.readthedocs.io/en/latest/). You may find some useful codes there. — André Pacheco, Jul 10 '19 at 15:57
@Berriel any ideas on how to write a lambda function for multiple expression as my gaussian blur function has 2 expressions, I thought for long but couldn't figure it out. — Beginner, Jul 15 '19 at 15:48
@Mark if you look at the code of [`transforms.Lambda(...)`](https://github.com/pytorch/vision/blob/8d580a1f087d1758cf181269d13ed717dba0c2bd/torchvision/transforms/transforms.py#L301-L316), you'll notice that you can pass any callable. Wrap your transformation in a function and just pass it. It should work. — Berriel, Jul 15 '19 at 16:51

score 0 · Accepted Answer · answered Jul 22 '19 at 12:47

This is how I did it-

The solution of the first part is first defining the required function and then calling in the transforms using the generic transforms in the following way-

def gaussian_blur(img):
   image = np.array(img)
   image_blur = cv2.GaussianBlur(image,(65,65),10)
   new_image = image - image_blur
   im = Image.fromarray(new_image)
return im

Solution of second part is to go through every image and calculate the mean and std deviation and then finally calling the mean and std deviation values in the transforms.-

   train_mean = []
   train_std = []

   for i,image in enumerate(train_loader,0):
       numpy_image = image[0].numpy()
       batch_mean = np.mean(numpy_image, axis=(0, 2, 3))
       batch_std = np.std(numpy_image, axis=(0, 2, 3))

       train_mean.append(batch_mean)
       train_std.append(batch_std)

   train_mean = torch.tensor(np.mean(train_mean, axis=0))
   train_std = torch.tensor(np.mean(train_std, axis=0))

    print('Mean:', train_mean)
    print('Std Dev:', train_std)

Final transform calling looks like this-

data_transforms = transforms.Compose([transforms.RandomCrop(512,512),
                                 transforms.Lambda(gaussian_blur),
                                 transforms.RandomRotation([+90,+180]),
                                 transforms.RandomRotation([+180,+270]),
                                 transforms.RandomHorizontalFlip(),
                                 transforms.ToTensor(),
                                 transforms.Normalize(mean=train_mean, std=train_std)
                               ])

Apply preprocessing to the dataset

1 Answers1