1

Normally I am using this kind of matlab code to do a projective transformation with a single set of reference points:

fixedpoints = [0 0; 50 0; 50 100; 0 100];
movingpoints= [752 361; 888 361; 885 609; 736 609];
transformationtype='projective';
tform = fitgeotrans(movingpoints,fixedpoints,transformationtype);
Imagepr = imwarp(Image,tform);

Now I have a photo that shows some (for example 10) jumbled paper sheets, each with known size, situated on a plain large table. The distances between these sheets are unknown.

So I would have 10 4-reference-point sets to obtain the projective transformation matrix and later the projected image of the 2d table surface.

How can I produce an optimized transformation matrix (and later a projected image), which would include all 4-points sets as a kind of best fit?

Thanks in advance.

peter e
  • 61
  • 4
  • "The distances between these sheets are unknown": do you mean that you don't know the fixed points ? –  Jan 28 '16 at 13:00
  • I meant that all sheets are positioned flat on the table surface and all have the same size (for example 30x20 cm). But the distance between each other and their rotation angles are unknown. I hope that helps for clarification. – peter e Jan 28 '16 at 14:25
  • With unknown positions and orientations, the problem is much harder. –  Jan 28 '16 at 14:26

1 Answers1

2

I feel the need to reformulate the problem as follows: you have a table with rectangular sheets of the same size* in arbitrary positions and an image taken by a camera in arbitrary position. You want to compute the homographic transform that maps the plane of the table to the image (or conversely). The coordinates of the four corners of every sheet are known in the image, but not on the table, and point correspondences are not available. The reference coordinate system of the table is arbitrary.

As hinted by the OP, the problem can be solved for every sheet independently by assigning "table" coordinates to the corners. The equations are linear. (Anyway two solutions are possible by swapping the long and short sides of the sheets.)

It appears to be impossible to combine the various per-sheet solutions, as the relative placement of the sheets is unknown, and the coefficients are unrelated.

Starting from an initial solution, one can think of using the least-squares approach to refine the solution, minimizing the total discrepancy between the known side lengths and those estimated by the transform. (We can use the known sides and diagonals, even though one of these six elements can be derived from the other five. The orientation ambiguity is solved by taking the orientation that gives the minimum discrepancy.)

Unfortunately, the equations aren't nice, calling for Levenberg-Marquardt or constrained minimization: sledge hammers.

The OP made an excellent suggestion: solve the projection problem for all sheets and compute the discrepancy metric for each and keep the solution with the lowest discrepancy. This reminds of RANSAC, where instead of random attempts, an exhaustive search is made**.

If one wants to further improve this solution (presumably an already pretty good one), one can think of derivative-less methods like the simplex (Nelder–Mead), Hooke-Jeeves patterns or sequential descent, trying to improve the discrepancy metric.

Taking the few best solutions has the benefit of giving an order of magnitude for the increments to apply to the coefficients. An extra difficulty appears, due to the fact that the "table" coordinate system isn't fixed. To avoid a drift of the solutions, one should add constrains, such as distances to the corners of the reference sheet.

*If the sheets are of different sizes, we can use the same approach if the sheet correspondence is known (but exact corner correspondence is unnecessary).

**If necessary, for increased robustenss, it should be possible to integrate an outlier rejection process in the minimization.

The projection of a plane onto the image is described by an homographic relation

X = (a x + b y + c) / (g x + h y + 1) 
Y = (d x + e y + f) / (g x + h y + 1)

There are eight unknown parameters, and it takes four point correspondences to determine them.

These equations are easily linearized as

a x + b y + c                 - g X x - h X y = X
                d x + e y + f - g Y x - h Y y = Y
  • I don't know how to do that. – peter e Jan 28 '16 at 14:17
  • Thanks for your fast answer, but I don't know how to do that. My idea is that I use the same fixedpoints for every sheet and just change the movingpoints-coordinates. so I obtain 10 different tform-matrices. They look all different because of the different rotation angles of the sheets on the table. How could I use the least-square method here? – peter e Jan 28 '16 at 14:33
  • AFAIK, achieving this is pretty difficult, technically speaking. Why do you want to globalize the transforms ? –  Jan 28 '16 at 14:39
  • Because it is difficult to measure the controlpoint-coordinates in an image with a high precision. The second problem is that the dimensions of the paper sheets are small in relation to the whole table surface, which I would like to project. In combination these two problems lead to low accuracy. I was hoping that there could be a higher accuracy with some kind of best fit method. – peter e Jan 28 '16 at 14:54
  • @petere if the sheets are small, you can treat them as points with a finite area. This area is proportional to the distance. There is an opportunity for another problem formulation. –  Jan 28 '16 at 15:54
  • thanks Yves but unfortunately that does not help me a lot... my new idea is that I will calculate the transformation-matrix for just one of these 4point-pairs and project the image. After that I try to detect all rectangles in the projected image and measure each legth/height/area. if these parameters do not coincide, I will have to retry with another rectangle for matrix calculation. So that would be some kind of an iterative approach....hopefully someone has a better idea. – peter e Jan 29 '16 at 13:10
  • You still don't realize that this won't work because all your transformations are completely unrelated. You are missing the world coordinates of the points. –  Jan 29 '16 at 13:16
  • in my eyes that new iterative approach will basically work beacause I use just one paper-rectangle for matrix calculation and projection. then I will detect all rectangles on that projected image and measure the rectangle sizes. So I can use for expample the variance of the recatangle sizes/areas as a quality/optimization parameter which has to be minimized. – peter e Jan 29 '16 at 13:48
  • That makes more sense to me now. I am adding to my answer. –  Jan 29 '16 at 14:12
  • @petere: I have completely reworded my answer. I do believe that your approach is an excellent one and should be preferred over least squares. It is likely that using the transform that gives the lowest residue is enough. And if you want to further lower it, relatively simple methods are available. –  Jan 30 '16 at 13:35
  • Thanks a lot for your professional and detailed answer. Maybe I will have some time the next days to realize the approach in a matlab code. – peter e Feb 01 '16 at 08:39
  • @petere: thanks for asking the question.I was very skeptical in the beginning. Now I am very confident that this works. I learnt something. –  Feb 01 '16 at 08:51