You cannot remove a number of a matrix in matlab, without changing its dimensions (if it's an square matrix, you'll get a column array). You have two options here:
1) (Option I'd choose) Transform your matrix to a cell by using cell2mat. And on each cell you'll have the value of the pixel in each channel ({..., [r g b], ...} in a cell) or in a channel ({... ;1, 1, 1; ...}). Then you can remove a position of the cell by putting it as empty;
yourCell{indexToRemove} = [];
2) Put a value to recognise the pixel as invalid. For instance, you can use black (0, 0, 0) or white (1, 1, 1) colour in that pixel index.
I can improve any of the two options, in case you want any of them.