2

I am new in python programming, so forgive me if my questions are too basic. I've been helped a lot by this forum before and thanks to you guys for all your contributions.

This time I have a set of 12,000 image data which I am performing singular value decomposition (svd) on and calculating their mean. Some of the images have pixels with very high positive or negative values which I don't want to use during computation, so I used numpy.ma.masked_array to exclude them from both svd and mean computation. And some images are smaller than others and they were padded with zeros values to make all images to have the same (pixel) dimension. But I also don't want the 'zero paddings' to be used during computation, so I used numpy.ma.masked_array to exclude them from both svd and mean calculation.

Here are some example images:

example beam images

The problem is that when I perform both svd and mean calculation, the masked values (array elements) are not excluded during computation. I have tried all that I know to resolve this without success. Below are the steps that I took.

from numpy.linalg import svd
import numpy as np
from numpy.ma import masked_array


n, x, y = images.data.shape
Z = []
meanimage = []

for icount in range(n):
    image = images[icount,:,:] # current image

    # creating a mask for too positively or negatively high values
    mask = (np.abs(image) > 2).astype(int); 
    yindex = 0; xindex = 0; 

    # --- creating a mask for zero padded values
    for i in range(y/2): # get the index of the first none zero pixel
        if image[i,x/2] != 0: 
            yindex = i
            break
    for i in range(x/2): # get the index of the first none zero pixel
        if image[y/2,i] != 0: 
            xindex = i
            break
    mask[:yindex,:] = 1;mask[-yindex:,:] = 1;
    mask[:xindex,:] = 1;mask[-xindex:,:] = 1;
    # --- 

    image = masked_array(images[icount,:,:], mask)

    Z.append(image.ravel())  # accummulating matrix for svd computation
    meanimage.append(image)  # accummulating matrix for for mean computation

# calc. SVD
u,s,v = svd(masked_array(Z))

#calc. mean image
meanimage = masked_array(meanimage).mean(axis=0)

bimage = np.dot(np.dot(u[:,:2],np.diag(s[:2])),np.transpose(v)[:2,:])

eigenimage = bimage[2,:].reshape(x, y)

The final results - eigenimage and meanimage - that I get does not exclude the masked values from computation. I don't know what I did wrong. Please, I need some ideas that will help me to resolve this.

Above are some samples of the images (beams) data that I am working with.

The final images that I get after computation for the eigenimage and meanimage are :

Eigen (beam) image (with SVD)

Eigen-image (beam)

Mean (beam) image (masked_array mean)

Mean image (beam)

From the above figures, both the eigenimage and meanimage loses a lot of side lobes information which are not desired.

But I was expecting the final eigen images to be like

Expected eigen image (beam)

Community
  • 1
  • 1
Kela
  • 41
  • 6
  • 2
    Masked arrays aren't yet supported for many linear algebra operations ([see here](http://docs.scipy.org/doc/numpy/reference/routines.ma.html#algebra) for a list of supported operators). Although the mean should work, SVD will not. One work-around would be to construct a boolean index for those pixels that you consider to be valid across all images, and perform SVD only on those. – ali_m Aug 02 '15 at 16:29
  • 2
    Generally only functions in the `np.ma` submodule (and masked arrarymethods) take the mask into account. Most other numpy code is not aware of the masking. A masked array is a regular array with a 'appended' mask and associated parameters. `svd` and `dot` just work with the regular array. – hpaulj Aug 02 '15 at 16:32
  • How about using the masked `filled` to put some sort of neutral value into the array? Depending on what you are trying to it could be `1`, `0`, `nan` or something else. How should masking affect the `svd`? – hpaulj Aug 02 '15 at 17:04
  • To communicate better what I want to archive, below are some 5 samples of the image data that I am working with (I have about 12,000 images). ![Figure 1]:/home/kela/Documents/work/sample_images.png In the figure above, I want to mask all zero pixels in each images which was used to pad them to make all images have the same dimension of 128 by 128, so during `svd` computation these padded pixels should not be used. Because with out masking them, they affect the final resultant eigen image. – Kela Aug 04 '15 at 19:01
  • Please @ali_m, throw more light on how I can construct (& use) “boolean indexes for valid pixels in all images” and how to instruct SVD to use only them for computation. – Kela Aug 04 '15 at 20:20
  • Could you edit your question to show the example image? Masking out pixels only really makes sense if the positions of the pixels you want to exclude are constant across images. – ali_m Aug 05 '15 at 02:20
  • @ali_m unfortunately my account don't yet have enough '**Reputation**' (only 6 but minimum of 10 required) to post images in this forum. I have used some sample images as my profile picture, since that's the closest I can get to upload an image. Hope you can download and view the images (5). The pixels that I want to mask out are not the same across images. Please, I am open to any other approach that will work better if using `masked_array` is not the best for this case. – Kela Aug 05 '15 at 13:00
  • Upload the image to imgur and post the URL - I can edit your question to include the image. It's not clear to me how invalid pixels *should* be treated when computing the SVD. If you're trying to perform PCA then it might make more sense to perform eigendecomposition on the sample covariance matrix, which can be straightforwardly computed with missing values. – ali_m Aug 05 '15 at 13:17
  • Some sample (beam) image data that I am working with can be found in the link [samples](http://i.imgur.com/JGfiM7d.png). For e.g, in the figure, the 1st beam (left to right) has information (data) mostly for the center lobe (& very little for the 1st side lobe) while the last beam has data up to the 2nd side lobe (although little). SVD computation should ignore invalid (& zero padded) pixels in each beam else the final eigen-beam will lose 1st & 2nd side lobes information and this is not desired. – Kela Aug 05 '15 at 16:47
  • Please @ali_m, help me with an e.g. illustrating how to use PCA to perform eigendecomposition on some samples (that computes with missing values) which I can follow to try the procedure on my data. – Kela Aug 05 '15 at 16:48

1 Answers1

1

The masked_array mean actually excludes masked pixels ('zero paddings') from mean computation. I confirmed this by comparing this result with the one calculated without a mask and noticed a remarkable difference, which confirms that numpy.ma.masked_array mean works perfectly for my case.

On the SVD Eigen image: The problem was with the transposing v (np.transpose(v)). I found out from documentation (1) that numpy.linalg.svd returns a transpose of v, so I just needed to perform the dot product without transposing v.

bimage = np.dot(np.dot(u[:,:2],np.diag(s[:2])),v[:2,:])

Community
  • 1
  • 1
Kela
  • 41
  • 6