Warp perspective and stitch/overlap images (C++)

Question

I am detecting and matching features of a pair of images, using a typical detector-descriptor-matcher combination and then findHomography to produce a transformation matrix.

After this, I want the two images to be overlapped (the second one (imgTrain) over the first one (imgQuery), so I warp the second image using the transformation matrix using:

cv::Mat imgQuery, imgTrain;
...    
TRANSFORMATION_MATRIX = cv::findHomography(...)
...
cv::Mat imgTrainWarped;
cv::warpPerspective(imgTrain, imgTrainWarped, TRANSFORMATION_MATRIX, imgTrain.size());

From here on, I don't know how to produce an image that contains the original imgQuery with the warped imgTrainWarped on it. I consider two scenarios: 1) One where the size of the final image is the size of imgQuery 2) One where the size of the final image is big enough to accommodate both imgQuery and imgTrainWarped, since they overlap only partially, not completely. I understand this second case might have black/blank space somewhere around the images.

score 5 · Accepted Answer · edited Jul 23 '21 at 16:45

5

You should warp to a destination matrix that has the same dimensions as imgQuery after that, loop over the whole warped image and copy pixel to the first image, but only if the warped image actually holds a warped pixel. That is most easily done by warping an additional mask. Please try this:

cv::Mat imgMask = cv::Mat(imgTrain.size(), CV_8UC1, cv::Scalar(255));
cv::Mat imgMaskWarped;
cv::warpPerspective(imgMask , imgMaskWarped, TRANSFORMATION_MATRIX, imgQuery.size());

cv::Mat imgTrainWarped;
cv::warpPerspective(imgTrain, imgTrainWarped, TRANSFORMATION_MATRIX, imgQuery.size());

// now copy only masked pixel:
imgTrainWarped.copyTo(imgQuery, imgMaskWarped);

please try and tell whether this is ok and solves scenario 1. For scenario 2 you would test how big the image must be before warping (by using the transformation) and copy both images to a destination image big enough.

edited Jul 23 '21 at 16:45

Gulzar

23,452
27
113
201

answered Aug 28 '15 at 08:37

Micka

19,585
4
56
74

Hmm... I am a bit confused. I am the code you wrote, but I obtain this: http://j.mp/warp001. (The two images with matches on them can be seen here: http://j.mp/warp002). What do you think? – Aug 28 '15 at 09:44
can you `cv::imwrite` `imgTrainWarped` and `imgQuery` right after warping please and share them to me? – Micka Aug 28 '15 at 09:59
After wrapping: imgQuery is http://j.mp/imgQuery , imgMaskWarped is http://j.mp/imgMaskWarped and imgTrainWarped is http://j.mp/imgTrainWarped – Aug 28 '15 at 10:10
Warping doesn't work as expected, maybe the warping must be done "the other way around"... can you add the original images, too? – Micka Aug 28 '15 at 10:12
I thought so, but I am not entirely familiar with the in-between Mask step, and I can't obtain the right thing. This is the original imgQuery: http://j.mp/imgQuery_original and this is the second image: http://j.mp/imgTrain Thank you very much for your time! – Aug 28 '15 at 10:27
but that means you warped `imgQuery` instead of `imgTrain`? Because in the warped image (before copying), the parts of imgQuery are visible?!? I mean your posted imgTrainWarped looks like your posted imgQuery translated to top. Instead it should look like imgTrain translated to top... – Micka Aug 28 '15 at 10:34
(There may have been a mistake with previous images as I was inadvertently running the code twice every time.) This is the original query image after warping (which doesn't affect it) and before copying: http://bit.ly/imgQuery_before , and this is query after copying: http://bit.ly/imgQuery_copied – Aug 28 '15 at 11:22
ok, this looks like you've transformed in the wrong direction... can you try `TRANSFORMATION_MATRIX.inv()` in all the warps? I'll work on a small example with images. – Micka Aug 28 '15 at 11:47
It works! Thank you so much for your time, Micka! Do you happen to know why a black line (at least in my case is black) appears when stitching? I am not sure whether it is due to my images or the program. – Aug 28 '15 at 11:54
probably the mask isn't transformed perfectly (so a little black border is added from the warped image), or there is a black line in the image. Instead of .copyTo(image, mask) it might be better to use some real BLENDING, like linear blending. – Micka Aug 28 '15 at 11:59
btw, since you had to invert the transformation: That means you have computed the homography in the wrong order (from query to train). In fact imho it makes more sense to compute the transformation from query to train, if the names resemble what they let assume, but then you have to compute the warp from query to train. – Micka Aug 28 '15 at 12:03
please try `cv::erode(imgMaskWarped, imgMaskWarped, cv::Mat());` before `.copyTo` call. This will reduce the mask by a small bit, so that a little bit less of the warped image will be copied to the original image. This will hopefully remove the black line, but if alignment wasnt perfect you will still see some artifacts – Micka Aug 28 '15 at 12:12

score 1 · Answer 2 · answered Aug 28 '15 at 11:38

1

Are you trying to create a panoramic image out of two overlapping pictures taken from the same viewpoint in different directions? If so, I am concerned about the "the second one over the first one" part. The correct way to stitch the panorama together is to cut both images off down the central line (symmetry axis) of the overlapping part, and not to add a part of one image to the (whole) other one.

answered Aug 28 '15 at 11:38

Strangled

29
6

I mean overlapping in the sense that one of the images remains untouched, while the other is plotted "on" it. Whether a previous step is to cut away the part of the first image (which is overlapped) or not would not affect the final result, I assume, so both procedures should do. I might be misunderstanding you... – Aug 28 '15 at 11:45
First of all, how were the images taken? If the camera was moved in parallel, such as aerial photos taken at different time moments in even horizontal flight, then disregard my comments. But if the camera was placed in a fixed point and just rotated somewhat around between the first and second shot, then the overlap will not be a perfect match _except_ on the symmetry axis of the overlapping part. – Strangled Aug 28 '15 at 11:51

score 0 · Answer 3 · answered Jul 20 '18 at 15:56

Accepted answer works but could be done easier with using BORDER_TRANSPARENT:

cv::warpPerspective(imgTrain, imgQuery, TRANSFORMATION_MATRIX, imgQuery.size(), INTER_LINEAR, BORDER_TRANSPARENT);

When using BORDER_TRANSPARENT the source pixel of imgQuery remains untouched.

score 0 · Answer 4 · answered Oct 21 '20 at 16:25

0

For OpenCV 4 INTER_LINEAR and BORDER_TRANSPARENT can be resolved by using cv::InterpolationFlags::INTER_LINEAR, cv::BorderTypes::BORDER_TRANSPARENT, e.g.

cv::warpPerspective(imgTrain, imgQuery, TRANSFORMATION_MATRIX, imgQuery.size(), cv::InterpolationFlags::INTER_LINEAR, cv::BorderTypes::BORDER_TRANSPARENT);

answered Oct 21 '20 at 16:25

Michael Lehning

11
1

Could you explain how the code you have posted is *significantly* different from that in the [answer by coyer](https://stackoverflow.com/a/51446320/10871073), posted over 2 years ago? – Adrian Mole Oct 21 '20 at 19:33

Warp perspective and stitch/overlap images (C++)

4 Answers4

Linked