How to compare if two images representing the same object if the pictures of the object belongs from two different sources - in OpenCV?

Question

Suppose I have an image of a car taken from my mobile camera and I have another image of the same car taken downloaded from the internet. (For simplicity please assume that both the images contain the same side view projection of the same car.)

How can I detect that both the images are representing the same object i.e. the car, in this case, using OpenCV?

I've tried template matching, feature matching (ORB) etc but those are not working and are not providing satisfactory results.

Can you put the results of ORB/template matching that you've tried to see why they are "not working" or "not providing satisfactory results"? — Rick M., Feb 28 '17 at 08:56
Consider accepting the best answer if you they were helpful. — Michał Gacka, Mar 10 '17 at 11:26

score 1 · Answer 1 · answered Feb 28 '17 at 06:19

1

SIFT feature matching might produce better results than ORB. However, the main problem here is that you have only one image of each type (from the mobile camera and from the Internet. If you have a large number of images of this car model, then you can train a machine learning system using those images. Later you can submit one image of the car to the machine learning system and there is a much higher chance of the machine learning system recognizing it.

From a machine learning point of view, using only one image as the master and matching another with it is analogous to teaching a child the letter "A" using only one handwritten letter "A", and expecting him/her to recognize any handwritten letter "A" written by anyone.

answered Feb 28 '17 at 06:19

Totoro

3,398
1
24
39

So if we replace the image of Car with that of a book cover, since the cover of a book is unique and if the same process is applied, do you have a suggestion for that? Like if I have mobile camera image of Dan Brown's inferno and an internet downloaded image of the same. What do you suggest in that case? – Neelesh Feb 28 '17 at 06:53
The text and illustration components in book covers make them a much better candidate for SIFT features. Book covers are generally planar, so the differences between the photo and the reference can be mostly represented using an affine transform, noise and lighting changes. SIFT works fairly well under these conditions. I have tried OpenCV SIFT example on book covers, just for testing. Works generally well. – Totoro Feb 28 '17 at 08:12
The paper at http://ai.pku.edu.cn/aiwebsite/research.files/collected%20papers%20-%20others/Book%20Cover%20Identification%20by%20Using%20Four%20Directional%20Features%20Filed.pdf is somewhat old, but the algorithm should be easy to implement in OpenCV. TinEye is a company that provides software to do this, you can have a look at their work too. – Totoro Feb 28 '17 at 08:12

score 1 · Answer 2 · answered Feb 28 '17 at 08:53

Think about how you can mathematically describe the car's features so that every car is different. Maybe every car has a different size of wheels? Maybe the distance from the door handle to bottom of the side window is a unique characteristic for every car? Maybe every car's proportion of front side window's to rear side window's width is an individual feature of that car?

You probably can't answer yes with 100% confidence to any of these quesitons. But, what you can do, is combine those into a multidimensional feature vector and perform classification.

Now, what will be the crucial part here is that since you're doing manual feature description, you need to take care of doing an excellent work and testing every step of the way. For example, you need to design features that will be scale and perspective invariant. Here, I'd recommend reading on how face detection was designed to fulfill that requirement.

Will Machine Learning be a better solution? Depends greatly on two things. Firstly, what kind of data are you planning to throw at the algorithm. Secondly, how well can you control the process.

What most people don't realize today, is that Machine Learning is not some magical solution to every problem. It is a tool and as every tool it needs proper handling to provide results. If I were to give you advice, I'd say you will not handle it very well yet.

My suggestion: get acquainted with basic feature extraction and general image processing algorithms. Edge detection (Canny, Sobel), contour finding, shape description, hough transform, morphological operations, masking, etc. Without those at your fingertips, I'd say in that particular case, even Machine Learning will not save you.

I'm sorry: there is no shortcut here. You need to do your homework in order to make that one work. But don't let that scare you. It's a great project. Good luck!

How to compare if two images representing the same object if the pictures of the object belongs from two different sources - in OpenCV?

2 Answers2