8

I have multiple images taken simultaneously pointing at the same direction from the same starting location. However, there is still a slight offset because these cameras were not in the exact same place when the picture was taking. I'm looking for a way to calculate the optimal translation/shear/skew/rotation needed to apply to match one image to another so that they overlay (almost) perfectly.

The images are in a .raw format which I am reading in 16 bits at a time.

I have been suggested (by my employer who is not a programmer [I'm an intern btw]) to take a portion of the source image (not at the edges) and brute-force search for a same-sized portion with a high correlation in data values. I'm hoping there is a less-wasteful algorithm.

andrw
  • 790
  • 1
  • 8
  • 20
  • If you *really* need to write this yourself, look up (for example) the [SIFT](http://www.cs.ubc.ca/~lowe/keypoints/) algorithm. Unless you really need to, though, you're probably better off using something like [Hugin](http://hugin.sourceforge.net/) or [CombineZP](http://www.hadleyweb.pwp.blueyonder.co.uk/CZP/News.htm) that can already do the job. – Jerry Coffin Jul 01 '11 at 00:08
  • 1
    The correct term for this operation is **image-registration**. You will probably be able to find an answer from [a search on SO](http://stackoverflow.com/search?q=image-registration). – rwong Jul 01 '11 at 06:18

3 Answers3

12

Here is a short code that does what you want (I use openCV 2.2):

  1. Suppose you have 2 images: srcImage,dstImage, and you want to align them
  2. The code is very simple. Use it as basis for your algorithm.

Code:

// Detect special points on each image that can be corresponded    
Ptr<FeatureDetector>  detector = new SurfFeatureDetector(2000);  // Detector for features

vector<KeyPoint> srcFeatures;   // Detected key points on first image
vector<KeyPoint> dstFeatures;
detector->detect(srcImage,srcFeatures);
detector->detect(dstImage,dstFeatures); 

// Extract descriptors of the features
SurfDescriptorExtractor extractor;  
Mat projDescriptors, camDescriptors;
extractor.compute(srcImage,  srcFeatures, srcDescriptors);
extractor.compute(dstImage , dstFeatures, dstDescriptors );

// Match descriptors of 2 images (find pairs of corresponding points)
BruteForceMatcher<L2<float>> matcher;       // Use FlannBasedMatcher matcher. It is better
vector<DMatch> matches;
matcher.match(srcDescriptors, dstDescriptors, matches);     

// Extract pairs of points
vector<int> pairOfsrcKP(matches.size()), pairOfdstKP(matches.size());
for( size_t i = 0; i < matches.size(); i++ ){
    pairOfsrcKP[i] = matches[i].queryIdx;
    pairOfdstKP[i] = matches[i].trainIdx;
}

vector<Point2f> sPoints; KeyPoint::convert(srcFeatures, sPoints,pairOfsrcKP);
vector<Point2f> dPoints; KeyPoint::convert(dstFeatures, dPoints,pairOfdstKP);

// Matched pairs of 2D points. Those pairs will be used to calculate homography
Mat src2Dfeatures;
Mat dst2Dfeatures;
Mat(sPoints).copyTo(src2Dfeatures);
Mat(dPoints).copyTo(dst2Dfeatures);

// Calculate homography
vector<uchar> outlierMask;
Mat H;
H = findHomography( src2Dfeatures, dst2Dfeatures, outlierMask, RANSAC, 3);

// Show the result (only for debug)
if (debug){
   Mat outimg;
   drawMatches(srcImage, srcFeatures,dstImage, dstFeatures, matches, outimg, Scalar::all(-1), Scalar::all(-1),
               reinterpret_cast<const vector<char>&> (outlierMask));
   imshow("Matches: Src image (left) to dst (right)", outimg);
   cvWaitKey(0);
}

// Now you have the resulting homography. I mean that:  H(srcImage) is alligned to dstImage. Apply H using the below code
Mat AlignedSrcImage;
warpPerspective(srcImage,AlignedSrcImage,H,dstImage.Size(),INTER_LINEAR,BORDER_CONSTANT);
Mat AlignedDstImageToSrc;
warpPerspective(dstImage,AlignedDstImageToSrc,H.inv(),srcImage.Size(),INTER_LINEAR,BORDER_CONSTANT);
DanielHsH
  • 4,287
  • 3
  • 30
  • 36
  • Wow, thank you! I'm taking a second (more like 100th) look into the documentation and it's been staring me in the face! Thanks for highlighting where I needed to look at more carefully and the great example code. – andrw Jul 01 '11 at 17:05
  • I can't seem to find the class FeatureDetector in my OpenCV installation (2.1), even though its in the documentation... any ideas? The first line: Ptr detector = new SurfFeatureDetector(2000); gives me : 1>.\makeTforms.cpp(34) : error C2065: 'FeatureDetector' : undeclared identifier (VS 2008) – andrw Jul 01 '11 at 18:20
  • I've reinstalled and I can't find that file anywhere. Did the installer change or something o_O – andrw Jul 01 '11 at 22:49
  • I am using opnencv 2.2. In there the file is at: C:\Program Files\OpenCV2.2\modules\features2d\include\opencv2\features2d. Also in OpenCV2.2\bin I have the opencv_features2d220.dll and in lib library the opencv_features2d220.lib. Are you using windows? If so, try to installing from here: http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.2/ – DanielHsH Jul 02 '11 at 10:37
  • Did you try to install from the link I posted? – DanielHsH Jul 03 '11 at 15:00
  • I have 11 linker errors when I install from the link provided and follow the installation instructions for VS 2008 – andrw Jul 03 '11 at 22:46
  • Really. Linker errors are good. At least not compilation errors. Did you add lib files to the project. I will add here another answer regarding the lib files, just in case.. – DanielHsH Jul 04 '11 at 05:08
  • I am glad to hear it. BTW, If you have problems with image matching and want to improve your algorithm, write me at DanielHsH at gmail dot com. I have experience with those algorithms – DanielHsH Jul 05 '11 at 19:01
0

Are the images taken standing from the same position but you're just rotated a bit so that they're not aligned correctly? If so then the images are related by a homography - i.e. a projective transformation. Given a set of correspondences between the images (you need at least 4 pairs), the standard way to find the homography is to use the DLT algorithm.

YXD
  • 31,741
  • 15
  • 75
  • 115
0

Avoid linker errors using the below code:

#include "cv.h"
#include "highgui.h"
using namespace cv;

// Directives to linker to include openCV lib files.
#pragma comment(lib, "opencv_core220.lib") 
#pragma comment(lib, "opencv_highgui220.lib") 
#pragma comment(lib, "opencv_contrib220.lib") 
#pragma comment(lib, "opencv_imgproc220.lib") 
#pragma comment(lib, "opencv_gpu220.lib") 
#pragma comment(lib, "opencv_video220.lib") 
#pragma comment(lib, "opencv_legacy220.lib") 

#pragma comment(lib, "opencv_ml220.lib") 
#pragma comment(lib, "opencv_objdetect220.lib") 
#pragma comment(lib, "opencv_ffmpeg220.lib") 

#pragma comment(lib, "opencv_flann220.lib") 
#pragma comment(lib, "opencv_features2d220.lib") 
#pragma comment(lib, "opencv_calib3d220.lib") 

// Your code here...
int main(void){
    Mat B = Mat:eye(3,3,CV_8U);
    return -1;
}
DanielHsH
  • 4,287
  • 3
  • 30
  • 36