0

I want to experiment with stereo imaging and therefore, installed two webcams which are aligned nearly parallel next to each other.

I detect corners on a charuco chessboard and calculate the fundamental matrix using stereoCalibrate from OpenCV. After that, I am calculating epiploar lines of the found pixel coordinates of the first image pair and visualize them using the code from the OpenCV tutorial.

The whole code is available here.

I ran this program multiple times, but what confuses me is that the fundamental matrix is everytime totally different.

One time, it is:

0.0 -0.00002 0.00411
0.00002 0.00000 -0.02593
-0.00624 0.02016 1

And another time, it is:

-0.00001 0.00004 0.01019
-0.00004 -0.00001 0.0887
0.00018 -0.08702 1

Further, one time the epipoles are within the image plane:

enter image description here

and another run, they are completely outside:

enter image description here

And each line should cross a dot since a line is an epiline and the dot is a found pixel in the other image. But this is also not the case.

As far as I understand, the epipoles should never change if the cameras are not moved, right? (which I don't do) Of course, for each run I use different images from my cameras so I understand that there should by a slight difference. But at least the epipoles should be at quite the same position. So do you have an idea why the fundamental matrix changes so drastically during runs?

Christoph Rackwitz
  • 11,317
  • 4
  • 27
  • 36
NMO
  • 748
  • 8
  • 16
  • 1
    **how many** pictures do you use per calibration? are you always holding the board at such an angle that it never is parallel to the image plane of any camera? when it's parallel, that's a useless angle, doesn't show "foreshortening". -- show all your data. and code. preferably easily browsable, not a zip file or anything. please review [mre]. – Christoph Rackwitz Feb 12 '22 at 19:26
  • Thank you for your tips. It took me quite some effort but I followed your suggestion and created a Github repo with all the code and data. I updated my questio accordingly. Can you have a look at it please? – NMO Feb 13 '22 at 11:01
  • if the camera's are parallel to each other then you are asking for problems as the values in the matrices will become very small & run into all sorts of floating point issues. Also you want as much overlap as possible [stereography works best when the cameras are at an angle](https://aaronolsen.github.io/tutorials/stereomorph/arranging_cameras_photography.html) – DrBwts Feb 13 '22 at 11:28
  • I would seriously question that assertion. cameras with parallel optical axes are _the norm_ in all stereo vision. the algorithms compensate for misalignment and warp the pictures to _be_ views with parallel axes, before block matching can take place. I also don't see any issue with numerics, from this standard arrangement. numerical issues do happen when the cameras aren't allowed to observe foreshortening. that article advises to use cameras at right angles... that would make block matching *impossible*. even feature matching will have trouble. – Christoph Rackwitz Feb 13 '22 at 14:04
  • I followed DrBwts suggestion and iincreased the distance between the cameras and the angle so that they look more "inwards" but as Christoph already assumed that didn't help. The F matrix values stay small and the reprojection error is high. – NMO Feb 13 '22 at 14:25
  • I've looked at the data and code... there must be something causing this, but I don't see it. all I can advise for now is to hold the board even more at an angle so foreshortening is more obvious – Christoph Rackwitz Feb 13 '22 at 14:26
  • Thank you Christoph for checking! One irritating fact which I just read in the OpenCV tutorial states: "Fundamental Matrix estimation is sensitive to quality of matches, outliers etc. It becomes worse when all selected matches lie on the same plane." See [here](https://answers.opencv.org/question/18125/epilines-not-correct/). Does that mean that the fundamental matrix should not be computed with a chessboard pattern? – NMO Feb 13 '22 at 14:55

0 Answers0