14

I am currently developing an application based on OpenCV/C++ to track small animals: Here is a example of the kind of video it should process. The program simply output the x,y position of the animal and the time for each area. This is graphical representation of the result onto the original.

My question is a bit awkward --- and maybe off topic --- in that I am not asking how to improve my program, but how to assess it. I am aware of the existence Bonn Benchmark on Tracking dataset, but it is not appropriate to my case.

The program is meant to process very long video, therefore, I cannot realistically ask independent humans to assess the position of the animals and compare human vs. program. I have also considered using robots or putting transponders on bigger animals in order to have the precise positions, but I do not really have the resources.

I came out with the idea of using a program to generate videos of blobs moving in a 2d environment. My question is simple:

Are you aware of any programmable high-level framework that I could use to graphically simulate the motion of an object moving stochastically in a parametrisable background ?

My dream would be to have a command line tool that would work like this:

$ program [BACKGROUND_OPTIONS] [OBJECT_OPTIONS] -V VIDEO_OUTPUT -P POSITIONS_OUTPUT

The background texture could be manipulated as well as the shape, colour and motion pattern of the object moving.

I know that I could probably "easily" make it myself (and I will if I cannot find anything), but I would prefer if the program assessed and the reference were as much independent as they can (for instance not both made by the same person).

User97693321
  • 3,336
  • 7
  • 45
  • 69
Quentin Geissmann
  • 2,240
  • 1
  • 21
  • 36
  • probably you can try Netlogo. It can be found here: http://ccl.northwestern.edu/netlogo/ – maths-help-seeker Sep 22 '12 at 15:59
  • @maths-help-seeker, Sorry for the late reply, Thank you. My colleague and I tried, but it results pretty much on programming our own agent based simulation. Since it is not a very complex problem, at the moment, we are investigating implementing our own benchmark tool using more standard programming tools. We will hopefully update the question with our results :D – Quentin Geissmann Sep 28 '12 at 19:54
  • It was only a suggestion. I am glad you tried it out! :D – maths-help-seeker Sep 30 '12 at 17:52
  • 3
    I generally abuse students for this purpose; make a gui in which random frames are served to the students in which they need to click on the bug's center of gravity. Let every student in the class process 1000 frames (say an hour of monkey work each). Bam, 30000 frames with ground truth annotated by independent people. Just to be sure, I serve them the occasional frame that I have annotated myself already, to see if they are taking the job seriously and attach a quality metric to their annotations. – dvhamme Oct 02 '12 at 07:38
  • Haha, good one. I will try to find slaves :P. – Quentin Geissmann Oct 02 '12 at 11:20
  • 1
    You can always resort to paying people too! I've used https://www.mturk.com/mturk/welcome for labeling ground truth for exactly this kind of tracking task in the past. It's a little bit of a project to get up and running and learn how to create tasks that will give you good data, but once you do you may find it useful for all sorts of things. And using real data instead of synthetic data will make your assessment of your algorithm much more believable. – Soverman Oct 05 '12 at 18:23
  • @Soverman Thank you, I did not know this can of offer existed... It is interesting – Quentin Geissmann Oct 06 '12 at 08:00
  • 1
    Isn't there any existing software solution in this problem space, with which you could analyse the same video then compare results? Then you can investigate the discrepencies in decreasing order of severity, fixing your program when it's worse. Of course, there's the risk that your program and the other one will make the same mistakes - a delta won't expose that. But, seems at least as promising as simulating inputs. – Tony Delroy Oct 19 '12 at 04:51
  • A HTML5 animation is enough? http://www.html5canvastutorials.com/kineticjs/html5-canvas-kineticjs-animate-position-tutorial/ – Industrial-antidepressant Oct 23 '12 at 00:24
  • Thank you, sorry for the delay, I am busy with another project. @Industrial-antidepressant We have considered this for quite sometime. It gives the advantage of being displayable on a website. On the other hand, it is going to be hard to do things like (tuning the noise, generate a chaotic background texture, record video...). We are now implementing a simple model in C++. This has the advantage to be integrable in the final application as a set of "example videos". – Quentin Geissmann Oct 24 '12 at 19:38
  • @Tony D Thank you, my colleague and I are developing this program because we are not satisfied at all by others in terms of precision. I also agree that simulating input is far from perfect, but it can maybe help us to see how the performance of the program changes as background and motion pattern are modified. We are still looking for alternative and eventually, we will need to compare our program to some of them anyway :D. I will try to implement it soon and will make it available. – Quentin Geissmann Oct 24 '12 at 19:43
  • Or do you want something like this: http://www.hasenkopf2000.net/wiki/page/animations-cairo-and-numpy/ – Industrial-antidepressant Oct 24 '12 at 20:54
  • This is pretty easy with matlab image processing. See http://youtu.be/RqAcJGUC6Pc, all i did was take a couple of images and make one do a random walk on the other. One could easily do multiple images, constrain their movement and movement statistics, or generate various kinds of noise for the background. Let me know if you want the Matlab source. – engineerC Oct 25 '12 at 18:32
  • It may be a hard task to fit to your case, but there is a free crowd simulation script for Blender (free 3D modeler) which is really flexible: http://www.harkyman.com/bp.html – Sga Oct 26 '12 at 10:16

1 Answers1

2

One thing I've seen several motion detection/tracking projects do is create test videos with some 3D rendering software such as Blender. It doesn't have the simple interface of your dream test creator, but it's a good testing tool for lots of reasons:

  • You can set up whatever scenario you want (varying perspective, number of objects, test length, motion paths, etc.)

  • You completely control lighting parameters, shapes, sizes, etc.

  • You can design simple tests to verify basic functionality (solid color background with solid colored moving spheres makes a good starting point), then branch into more complex scenarios (other static objects, objects occluding other objects, background images, and so on).

Matt Kline
  • 10,149
  • 7
  • 50
  • 87
  • Thank you, it seems to be a very good solution... In practice, I will have to generate many video and change the variables I mentioned. Does Blender provide natively a scripting mode ? I do not intend to track more than one agent, but I am particularly interested by object occlusion for instance... As I said before, I am quite busy at the moment, but I will give a go to blender. Can you think of a good tutorial ? – Quentin Geissmann Oct 27 '12 at 09:18
  • 1
    Blender has full [python support](http://wiki.blender.org/index.php/Doc:2.4/Manual/Extensions/Python), so I would think you'd be able to script a test generator using that. The previous link contains a scripting example, and general tutorials for Blender can be found [here](http://www.blender.org/education-help/tutorials/). – Matt Kline Oct 27 '12 at 19:42
  • I have been waiting for a good motivation to learn python for a while... It could be it :D – Quentin Geissmann Oct 28 '12 at 10:30