7

I am just starting to explore OpenCV and the EmguCV .NET wrappers for it and need some general direction from folks who understand the big picture of its capabilities and perhaps those who have tackled a task similar to what I need to accomplish.

I will have a series of still photos and in each image will appear an object or nothing. The objects are pieces of metal hardware (bolts) and will be laying on their side with their length parallel to the top/bottom of the image (i.e., the picture is taken from above). If there is an object, it will be one of about 100 discrete types of bolt, some with very similar though not identical features and dimensions. For example, they will all be mostly rectangular in profile but will vary in length and width (diameter) and can have heads that are either hexagonal or round (which will be seen in profile as rectangular or as the minor segment of a circle, respectively) or will have conical heads for countersunk applications. An illustration of the types of parts I'm talking about (this is just to show the types of parts - my images are photos of single parts):

An Illustration
(source: donsnotes.com)

I need to classify them such that all sizes and types are differentiated. A 1-3/8" bolt should classify as different from a 1-1/2" bolt even if they are the same diameter and have the same head type. The minimum length difference between sizes will be 1/8", not the standard 1/16".

If it matters, I have good control of the following:

  • Lighting (but back-lighting isn't going to be practical)
  • The appearance of the background (useful for background subtraction maybe?)
  • The distance from the camera to the object (identical objects will always appear to be the same size in the images)
  • Generally, the position of the bolt - it will laying horizontally on its side, parallel with the top/bottom edges of the image frame. I cannot control whether its head is to the left or right in the image.

Unfortunately I cannot find any online papers or articles that directly address what I need to do - but lots that illustrate simpler tasks like finding a colored ball or finding rectangles. I can't find anything on identifying and classifying each of a large(ish) number of different but similar shapes. I do have two of the suggested OpenCV books and while they are great, they don't seem to address this issue.

I have found pretty clean Canny edges on my sample images but there is lots of noise in the interior of the part due to lighting. This makes finding clean Hough line segments kind of spotty.

I'm not certain if I should try to pare down the list of possible matches using absolute dimensions calculated by measuring across the Canny edges - then use something more robust like a cascade classifier...? Or what.

I'm really just looking for someone's opinion on a general strategy or a point in the right direction...

Can anyone give me something to start trying? I'm really at a loss.

Thanks!

Glorfindel
  • 21,988
  • 13
  • 81
  • 109
Matt Redmond
  • 726
  • 7
  • 12
  • 1
    Hum, this sounds like a nightmarish 3 man/months (or more) software development project if done purely with a camera. Can't you get additional signals? A scale with digital readout is cheap these days, and it'd probably pare down the search space a lot. Ya know, Second Law of Computer Vision: "There ain't a problem of Computer Vision that can't be solved by an adequate amount of hardware" ;-) – Francesco Callari Mar 20 '13 at 01:04
  • Consider adding additional features other than just the image features to "weed" out what an instance may/may not be. Something that comes to my mind is weight. But like others have stated, you are the expert in this domain so you will need to be the one who comes up with the high-level distinguishing features. Unfortunately pure image classification is going to be very difficult unless you can eliminate most of the aforementioned image noise (such as lighting). – Porkbutts Mar 20 '13 at 22:29
  • Thanks, guys, for your comments. I wish I could weigh them but they are in [fast] motion. It was enough to get the lighting and shutter speed right. Stopping them for weighing isn't going to be practical and I'm reasonably certain I can't afford anything that would reliably weigh them in motion! – Matt Redmond Mar 21 '13 at 17:24

2 Answers2

4

From my experience, I would suggest you the following. You know better than most of us how to classify manually, right? So think what happens in your brain while you classify. Suppose, you see a circular shape from the top view, so now your problem becomes how to identify that shape? Post such problems here, it will be lot easier for other people. Feature extraction is nothing but what brain thinks when you see that object.

For this question, I would suggest you first keep a dark background so that it will become necessary to binarize that image. Then if its a top view, you can see either a circle, hexagon etc. Then get only edges. Then get a minimal bounding circle and get its diameter. Algorithm and code to find minimum bounding circle can be obtained here.

As far as shape is concerned, I suggest you take gradient of that binary image and then calculate the angle of that gradients only at the edge points (which you got in top view). Histogram of that gradients will be your feature vector. See the bar plot of different shapes. If your brain can distinguish, then you can think about which classifier to use. I would rather not comment on that now since it depends on lot of things, such as distribution of features, their separability and most importantly speed requirement. But don't worry about classifier now.

Now, lets try to tackle the height. I assume you can get the front view, then just calculate the bounding box (you can get that from regionprops function in MATLAB).

Note that I am just imagining and talking all these things. You have to first do what I said in the first paragraph and then see if other part of the answer makes sense to you. I assume you have certain mathematical background to understand certain terminologies in this answer. If not, please do not hesitate to ask.

P.S. +1 for a good question.

Autonomous
  • 8,935
  • 1
  • 38
  • 77
  • 1
    Thanks very much for your reply. I am going to work on a minimal bounding rectangle first. Hopefully from that I can reliably determine the part length. If I then crop the image in the middle of that rectangle I should be dealing with only a portion of the bolt shank & can determine its width. At that point I'm halfway home - will work on the more complex problem of classifying the head shape last. Hopefully will gain enough experience during part one that I'll have fewer questions when working on the heads. Thanks again! – Matt Redmond Mar 21 '13 at 17:19
  • shape descriptor which was mentioned above has an opensource code too. – Autonomous Mar 21 '13 at 19:36
4

Your problem is a typical pattern recognition task. In your scenario I would use a shape descriptor and then use it for recognition. An interesting approach is described in paper "Shape Matching and Object Recognition using Shape contexts".

One of the authour Jitendra Malik is the teacher of an interesting coursera computer vision course I attend a couple of months ago.

Luca Del Tongo
  • 2,702
  • 16
  • 18
  • Thanks for that - it looks like a very helpful paper but a lot to get my head around. I think it will apply to the bolt head classification portion of what I'm doing. I really appreciate your reply! – Matt Redmond Mar 21 '13 at 17:22