How to apply iOS VNImageHomographicAlignmentObservation warpTransform?

Question

I'm testing Apple's Vision Alignment API and have questions regarding VNHomographicImageRegistrationRequest. Has anyone got it to work? I can get the warpTransform out of it, but I've yet to see a matrix that makes sense, meaning, I'm unable to get a result that warps the image back onto the source image. I'm using Opencv warpPerspective to handle the warping.

I'm calling this to get the transform:

class func homography(_ cgImage0 : CGImage!, _ cgImage1 : CGImage!, _ orientation : CGImagePropertyOrientation, completion:(matrix_float3x3?)-> ())
{
let registrationSequenceReqHandler = VNSequenceRequestHandler()
let requestHomography = VNHomographicImageRegistrationRequest(targetedCGImage: cgImage1, orientation: orientation)
let requestTranslation = VNTranslationalImageRegistrationRequest(targetedCGImage: cgImage1, orientation: orientation)

do
{
    try registrationSequenceReqHandler.perform([requestHomography, requestTranslation], on: cgImage0)  //reference

    if let resultH = requestHomography.results?.first as? VNImageHomographicAlignmentObservation
    {
        completion(resultH.warpTransform)
    }

    if let resultT = requestTranslation.results?.first as? VNImageTranslationAlignmentObservation
    {
        print ("translation : \(resultT.alignmentTransform.tx) : \(resultT.alignmentTransform.ty)")
    }
}
catch
{
    completion(nil)
    print("bad")
}

}

This works and outputs a homography matrix, but its results are drastically different than what I get when I do SIFT + Opencv findHomography (https://docs.opencv.org/3.0-beta/doc/tutorials/features2d/feature_homography/feature_homography.html)

Regardless of my image pairs, I'm unable to get reasonable homographic results from the Apple Vision dataset.

Thanks in advance,

I tried this as well, and was unable to get a result that was close to OpenCV. Maybe it's an apple bug — blueether, Oct 14 '18 at 20:38

Jan-Michael Tressler · Answer 1 · 2019-12-13T01:22:23.970

0

For future reference, I was able to correlate Apple's homography matrix with Opencv's matrix. Basically, the Core Image's image origin is the bottom left-hand corner of the image. Opencv's origin is the top left-hand corner. To convert Core Image's homography matrix to Opencv coordinates, one needs to apply the following transform:

H_opencv = Q * H_core_image * Q

where Q = [1 0 0; 0 -1 image.height; 0 0 1]

Update with Note:

I defined Q as a row-major matrix.

Apple's simd matrices are column-major. Opencv's matrices are row-major. To get the above equation to work, you may have to use the transpose of Q.

edited Dec 13 '19 at 01:22

answered Oct 16 '18 at 22:51

Jan-Michael Tressler

629
5
17

I am using VNHomographicImageRegistrationRequest warp transform matrix and opencv’s warpPerspective to transform overlapping images. So after multiplying the matrix with Q (y-flipped matrix) I was not able to get close to the desired opencv matrix .So I did some digging, the problem statement is defined as a change of bases matrix where both matrices (opencv and iOS homography matrix) should be similar. So the 1st property where both determinants should be almost equal failed. And even the trace was not close enough.How do I correlate these two matrices if they are mutually exclusive? – Manoj Arun S Nov 29 '19 at 06:17
Both matrices are similar after you apply the Q transform (for the same coordinate system). However, since homography is a combination of rotation, shear, and scaling, the two matrices should only be equal in how they modify points within the image frame. I would look at https://stackoverflow.com/a/51613151/6693924 on how to apply the results from VNHomographicImageRegistrationRequest to the two images. You can then use openCV tutorial using the Q * H_iOS * Q matrix to get the exact same results in OpenCV. – Jan-Michael Tressler Nov 29 '19 at 18:13
Jan-Michael , Thanks for your reply. I have tried with the solution that's provided in another SO post but that also doesn't work as expected for the alignment. Here's the output of the image - https://ibb.co/GFt206q and also my code snippet here - https://pastebin.com/u5FvfMrP . Can you kindly check and advise your inputs. Thanks in Advance. Here's the reference and floating image Reference - https://ibb.co/Ny614hc Floating - https://ibb.co/vY5rmtb – Manoj Arun S Dec 03 '19 at 12:23
@ManojArunS I just took a look at your code. From what I see, you don't need to adjust the homography by Q if you're using the iOS core image library. The Q matrix is used when one wants to warp the image via opencv (i.e. cv2.warpPerspective). I'll see if I can find my old Xcode playground later today or tomorrow. I'll use your images. – Jan-Michael Tressler Dec 03 '19 at 18:51
Thank you Jan. Looking forward to it. But i have used the approach that you have mentioned and it doesn't work. – Manoj Arun S Dec 04 '19 at 12:33
@ManojArunS what is the output when you remove Q from your calculations? Are the images shown above with Q or without Q? Here are the steps I would take to troubleshoot this: First, calculate Homography in OPENCV. Second, get an output in OPENCV using warpPerspective. Third, after you are satisfied, convert the Homography in OPENCV to iOS using Q. Verify the warped image in iOS matches exactly to that from OPENCV. Forth, then use iOS homography to see if the results are similar. Let me know how it goes. – Jan-Michael Tressler Dec 04 '19 at 19:32
Here's the matrix without Q for the homography and opencv matrix. https://pastebin.com/0vbZcUAc The images shown above are without Q mat. I have already tried the approach that you have mentioned above. Kindly let me know if you need any detail information. Thanks. – Manoj Arun S Dec 05 '19 at 14:32

How to apply iOS VNImageHomographicAlignmentObservation warpTransform?

1 Answers1

Linked