Android: Images alignment

Question

I need to align different images in my android application, using the OpenCV library. I found a solution in this thread.

public static Bitmap alignImagesHomography(Bitmap A, Bitmap B)
{
   final int warp_mode = MOTION_HOMOGRAPHY;
   Mat matA = new Mat(A.getHeight(), A.getWidth(), CvType.CV_8UC3);
   Mat matAgray = new Mat(A.getHeight(), A.getWidth(), CvType.CV_8U);
   Mat matB = new Mat(B.getHeight(), B.getWidth(), CvType.CV_8UC3);
   Mat matBgray = new Mat(B.getHeight(), B.getWidth(), CvType.CV_8U);
   Mat matBaligned = new Mat(A.getHeight(), A.getWidth(), CvType.CV_8UC3);
   Mat warpMatrix = Mat.eye(3, 3, CV_32F);
   Utils.bitmapToMat(A, matA);
   Utils.bitmapToMat(B, matB);
   Imgproc.cvtColor(matA, matAgray, Imgproc.COLOR_BGR2GRAY);
   Imgproc.cvtColor(matB, matBgray, Imgproc.COLOR_BGR2GRAY);
   int numIter = 5;
   double terminationEps = 1e-10;
   TermCriteria criteria = new TermCriteria(TermCriteria.COUNT +  TermCriteria.EPS, numIter, terminationEps);
   findTransformECC(matAgray, matBgray, warpMatrix, warp_mode, criteria, matBgray);
   Imgproc.warpPerspective(matA, matBaligned, warpMatrix, matA.size(), Imgproc.INTER_LINEAR + Imgproc.WARP_INVERSE_MAP);
   Bitmap alignedBMP = Bitmap.createBitmap(A.getWidth(), A.getHeight(), Bitmap.Config.RGB_565);
   Utils.matToBitmap(matBaligned, alignedBMP);
   return alignedBMP;
}

public static Bitmap alignImagesEuclidean(Bitmap A, Bitmap B)
{
   final int warp_mode = MOTION_EUCLIDEAN;
   Mat matA = new Mat(A.getHeight(), A.getWidth(), CvType.CV_8UC3);
   Mat matAgray = new Mat(A.getHeight(), A.getWidth(), CvType.CV_8U);
   Mat matB = new Mat(B.getHeight(), B.getWidth(), CvType.CV_8UC3);
   Mat matBgray = new Mat(B.getHeight(), B.getWidth(), CvType.CV_8U);
   Mat matBaligned = new Mat(A.getHeight(), A.getWidth(), CvType.CV_8UC3);
   Mat warpMatrix = Mat.eye(2,3,CV_32F);
   Utils.bitmapToMat(A, matA);
   Utils.bitmapToMat(B, matB);
   Imgproc.cvtColor(matA, matAgray, Imgproc.COLOR_BGR2GRAY);
   Imgproc.cvtColor(matB, matBgray, Imgproc.COLOR_BGR2GRAY);
   int numIter = 5;
   double terminationEps = 1e-10;
   TermCriteria criteria = new TermCriteria(TermCriteria.COUNT + TermCriteria.EPS, numIter, terminationEps);
   findTransformECC(matAgray, matBgray, warpMatrix, warp_mode, criteria, matBgray);
   Imgproc.warpAffine(matA, matBaligned, warpMatrix, matA.size(), Imgproc.INTER_LINEAR + Imgproc.WARP_INVERSE_MAP);
   Bitmap alignedBMP = Bitmap.createBitmap(A.getWidth(), A.getHeight(), Bitmap.Config.RGB_565);
   Utils.matToBitmap(matBaligned, alignedBMP);
   return alignedBMP;
}

public static Bitmap alignExposures(Bitmap A, Bitmap B) {
    Mat matA = new Mat(A.getHeight(), A.getWidth(), CvType.CV_8UC3);
    Mat matB = new Mat(B.getHeight(), B.getWidth(), CvType.CV_8UC3);
    Utils.bitmapToMat(A, matA);
    Utils.bitmapToMat(B, matB);

    List<Mat> src = new ArrayList<>();
    src.add(matA);
    src.add(matB);
    Bitmap output = Bitmap.createBitmap(A.getWidth(),A.getHeight(), Bitmap.Config.RGB_565);
    AlignMTB align = createAlignMTB(8, 4, false);
    align.process(src,src);
    for(int i = 1; i < src.size(); i++) {
        add(src.get(0),src.get(i),src.get(0));
    }
    Utils.matToBitmap(src.get(0),output);
    return output;
}

I tried all the three methods written by the user wegenerEDV. Anyway, the first two methods return the same picture as the "Bitmap A" given as input; the third method actually aligns the pictures, but the resulting image is overexposed:

original: https://i.stack.imgur.com/YnwYH.jpg

aligned: https://i.stack.imgur.com/ka8kT.jpg

Has anybody found a different solution? Or do these methods actually work and I am doing something wrong?

The best solution to me is to correct the alignImagesHomography method. It actually does something, because it takes around 30 seconds to process the final picture, but then it is exactly equal to the input image.

Answer 1

I've never used findTransformECC(), used by your first and second methods, and I'm not familiar with that algorithm. The only difference between those two methods is the type of transform findTransformECC() is asked to find; homographic transforms are a superset of Euclidean transforms so the first method (using MOTION_HOMOGRAPHY) would be most robust for your use case, though it might also be slower.

the first two methods return the same picture as the "Bitmap A" given as input

If these two methods are working correctly, the result should look almost identical to Image A, even though it is produced from pixels of Image B. Have you checked that the result is bitwise identical to Bitmap A, and does not just look similar? I think I can see the same bug in both methods: findTransformECC() finds a mapping from matBgray onto matAgray (see the docs), but warpPerspective() and warpAffine(), respectively, are used to apply the resulting transform to matA, storing the result in matBaligned; they should be applied to matB. The only way I can see that you would get a result that is bitwise identical to Image A is if your homography calculation fails, so that the result homography ("warpMatrix") still contains its initial state, which is the identity matrix. The identity homography, incorrectly applied to matA (because of the above bug), will of course give another exact copy of matA in matBaligned. You could check this by printing warpMatrix after calculation and seeing if it is the identity matrix. You should also add some error checking, because at the moment you don't know whether the methods you call are failing at all, or why (eg. bad input parameters, unable to find any correspondences, etc etc).

Your third method, alignExposures(), uses AlignMTB which is intended for HDR imaging, and I don't know how it does alignment. It may only handle 2D translation. The loop in that method is adding the output images back onto one of the source images, so it will saturate to white. If the result you want is the aligned images averaged together (is that what you want?) you should create a new output matrix with a larger datatype (eg. CV_16UC3) to acclumulate the result images in and calculate the average, then use cv::convertTo() to reduce that buffer back to the original datatype (eg. CV_8UC3).

Another algorithm I have used successfully for homographic alignment is:

• Create a feature detector, eg. SIFT, from the features2d or xfeatures2d library. I used AKAZE, because SIFT is patented.
• Detect features and descriptors in both your images using detectAndCompute().
• Create a BFMatcher and perform brute force matching of feature points from both images using match().
• If you want to at this point, you can print the matching feature points, draw them onto the images and view them, etc etc.
• Organize the found correspondences into two lists (source and destination points) suitable for findHomography().
• Call findHomography() using the RANSAC flag.
• Apply the homography using warpPerspective().

With this method there are more opportunities to inspect and validate your data, which will help with debugging.