4

I had an idea for which I need to be able to recognize certain objects or models from a rendered three dimensional digital movie.

After limited research, I know now that what I need is called feature detection in the field of Computer Vision.

So, what I want to do is:

  1. create a few screenshots of a certain character in the movie (eg. front/back/leftSide/rightSide)
  2. play the movie
  3. while playing the movie, continuously create new screenshots of the movie
  4. for each screenshot, perform feature detection (SIFT?, with openCV?) to see if any of our character appearances are there (they must still be recognized if the character is further away and thus appears smaller, or if the character is eg. lying down).
  5. give a notice whenever the character is found

This would be possible with OpenCV, right?

The "issue" is that I would have to learn c++ or python to develop this application. This is not a problem if my movie and screenshots are applicable for what I want to do.

So, I would like to first test my screenshots of the movie. Is there a GUI version of OpenCV that I can input my test data and then execute it's feature detection algorithms manually as a means of prototyping?

Any feedback is appreciated. Thanks.

Tom
  • 8,536
  • 31
  • 133
  • 232

5 Answers5

4

There is no GUI of OpenCV able to do what you want. You will be able to use OpenCV for some aspects of your problem, but there is no ready-made solution waiting there for you.

While it's definitely possible to solve your problem, the learning curve for this problem is quite long. If you're a professional, then an alternative to learning about it yourself would be to hire an expert to do it for you. It would cost money, but save you time.

EDIT

As far as template matching goes, you wouldn't normally use it to solve such a problem because the thing you're looking for is changing appearance and shape. There aren't really any "dynamic parameters to set". The closest thing you could try is have a massive template collection that would try to cover the expected forms that your target may take. But it would hardly be an elegant solution. Plus it wouldn't scale.

Next, to your point about face recognition. This is kind of related, but most facial recognition applications deal with a controlled environment: lighting, distance, pose, angle, etc. Outside of that controlled environment face detection effectiveness drops significantly. If you're detecting objects in a movie, then your environment isn't really controlled.

You may want to first try a simpler problem of accurately detecting where the characters are, without determining who they are (video surveillance, essentially). While it may sound simple, you'll find that it's actually non-trivial for arbitrary scenes. The result of solving that problem may be useful in identifying the characters.

mpenkov
  • 21,621
  • 10
  • 84
  • 126
  • My lack of knowledge caused me to believe that a simple call to testTemplate(source, template, ...) with some dynamic parameters would do the trick. It seems like there is much more involved though. However, I also figured that what I want is nearly the same as Face Recognition, isn't it? There surely must be more out there regarding this type of object recognition? – Tom Jan 31 '11 at 08:30
  • I've tried to answer your questions in an edit. I'm in a bit of a rush now, so if I've missed something let me know and I'll come back to it. – mpenkov Jan 31 '11 at 09:05
  • 1
    Thanks. It is not quite the answer I was hoping for though ;) Assuming that there really is nothing out there on the web (my searches would comply with this assumption) that describes a compareable situation as mine, I guess I would have to start from scratch and really dig into this field of Computer Vision. It seems like a highly sophisticated and rather complex task, so I might just have to get my mind off this "little plan". – Tom Jan 31 '11 at 09:59
  • 1
    If you're genuinely interested in computer vision, then don't let my pessimism stop you. It's a fun and exciting field to work in. The problem is most of the tools out there aren't ready-to-use out-of-the-box solutions, and it's incredibly difficult to use them if you don't understand how they work. In a way, that's part of the fun. – mpenkov Jan 31 '11 at 13:31
3

There is Find-Object by Mathieu Labbé. It was very helpful for me to start getting an understanding of the descriptors since you can change them while your video is running to see what happens.

This is probably too late, but might help someone else looking for a solution.

KobeJohn
  • 7,390
  • 6
  • 41
  • 62
2

Well, using OpenCV you would of taking a frame of a video file and do any computations on it.

You can do several different methods of detecting a character on that image, but it's not so easy to have it as flexible so you can even get that person if it's lying on the floor for example, if you only entered reference images of that character standing.

Basically you could try extracting all important features from your set of reference pictures and have a (in your case supervised) learning algorithm that gets a good feature-vector of that character for classification.

You then need to write your code that plays the video and which takes a video frame let's say each 500ms (or other as you desire), gets a segmentation of the object you thing would be that character and compare it with the reference values you get from your learning algorithm. If there's a match, your code can yell "Yehaaawww!" or do other things...

But all this depends on how flexible you want this to be. You could also try a template match or cross-correlation which basically shifts the reference image(s) over the frame and checks how equal both parts are. But this unfortunately is very sensitive for rotation, deformations or other noise... so you wouldn't get that person if its i.e. laying down. And I doubt you can get all those calculations done in realtime...

Basically: Yes OpenCV is good to use for your image processing/computer vision tasks. But it offers a lot of methods and ways and you'd need to find a way that works for your images... it's not a trivial task though...

Hope that helps...

florianbaethge
  • 2,520
  • 3
  • 22
  • 29
  • So I cannot test whether my screenshots are actually applicable for the task? Is there no other software that could perform these tests for me, before I start creating such a time investing application? – Tom Jan 30 '11 at 23:13
  • 1
    Well you could upload one or two example frames of your videos so we can see what it looks like and whether it would be a hard task to extract the important features of the characters... You're probably not going to find a ready software that does that for you... anyway it's going to be a time-consuming task, all we can do is help you get there... ;-) – florianbaethge Jan 30 '11 at 23:25
1

Have you tried computer vision frameworks like Cassandra? There you can exactly do that just by some mouse clicks.

Steffen Roeber
  • 127
  • 7
1

Have you tried looking at some of the work of the Oxford visual geometry group?

Their Video Google system describes to a large extent what you want, instance detection.

Their work into Naming People in TV shows is also pretty relevant. A face detection and facial feature pipeline is included that can be run from Matlab. Are you familiar with Matlab?

peakxu
  • 6,667
  • 1
  • 28
  • 27