I am trying to emulate the functionality of alpha compositing in Matlab or to be more specific the compose types CopyOpacity and Over from ImageMagick. First goal is to mask out a region with heavy aliasing edges like the black regions in the images in the set (image set with aliasing edges). This should be done with a compatible gray scale mask were black regions should be eliminated and white should be preserved see gray scale mask. Important is the continuous transition between black and white resulting in a continuous transition between transparent and opaque, see screenshot with transparent regions).
There has been two approaches to compose a foreground FG with and background BF with a gray scale mask CM_mask_blur_alpha but the results were not as expected (see an image set with post-processing with ImageMagick as reference for a pure Matlab procedure).
1) In the first code the gray scale mask is unexpectedly treated as a binary mask resulting in unacceptable aliasing effects at the edges of the formerly black regions (image set for first approach):
FG = uint8(CM_mask_blur_alpha .* FG + (1 - CM_mask_blur_alpha) .* BG);
2) The second approach lead to a visible continuous transition between a foreground and the background but there are aliasing effects from the FG remaining (image set for second approach):
FG = uint8(bsxfun(@times, CM_mask_blur_alpha, FG) + bsxfun(@times, (1 - CM_mask_blur_alpha), BG));
It seems that a one-step approach is not working so I am looking for a two-step approach like in ImageMagick with a masking out of the black regions resulting in an intermediate image with transparency and as second step a compose over of this intermediate image over an background. It is partly a problem setting as in MATLAB: Applying transparent mask over an RGB image and blending with another but there is no gray scale mask and I could not adapt solution parts like the generation of an alpha channel with values derived from the gray mask.