5

I am trying to implement an application that projects an image onto a page of a notebook, using OpenCV, a webcam and a projector. To achieve that, I am doing the following steps:

  1. I am using a webcam to detect the four corners points of a page.
  2. A homography is learned between the four corner points of the camera image and their projections on my desk, as seen in the camera. By using the inverse transformation, I will be able to know where I should draw something on my camera image, so that the projection "ends up" at a desired location.
  3. I am applying the inverse transformation to the detected four corners points of the page.
  4. I am warping the desired image to the new, transformed set of points.

So far it works well, if the notebook is on my desk and wide open. Like in this picture:

enter image description here

But if I try to close one side (or both), the following happens:

enter image description here

See the problem? In the first picture the image is perfectly aligned with the edges of the page and remains so if you rotate or translate the notebook, while keeping it on the desk. But that doesn't happen in the second image, where the the top edge of the image is no longer parallel to the top edge of the page (the image becomes more and more skewed).

Can anyone explain why I get this projection problem or at least point me to some resources where I can read about it? I need to mention that the projector and the webcam are placed above and to the left of the notebook, not right above them.

Any tips or suggestions are welcome. Thank you!

Lucas Zamboulis
  • 2,494
  • 5
  • 24
  • 27
Mihai Morariu
  • 166
  • 1
  • 7
  • Interesting job :). Well, which corners did your programm detect, the four corners of the full size notebook or the four corners of the half-size notebook? – rookiepig Apr 02 '14 at 12:20
  • Let's say we are interested in projecting onto one page, so the four corners of the right page. – Mihai Morariu Apr 02 '14 at 12:25
  • I don't understand what you are doing in step 2. You are talking about a 'camera image', is this the webcam image or the image to be projected ? Also, why do you consider the projections of the corner points on your desk if you want to project on the page of the notebook ? – BConic Apr 02 '14 at 12:43
  • Sorry, maybe that step is indeed not clear enough. The "camera image" is the webcam image. Regarding the second question, when you say "corner points", do you refer to the corner points of the notebook's page or the corner points of the camera (webcam) image? – Mihai Morariu Apr 02 '14 at 13:01
  • In other words, I don't understand what you mean by 'their projections on my desk, as seen in the camera' in your 2nd step. And by the way, it seems like you're assuming the camera is also the projector, since you never mention the projector in your 4 steps. – BConic Apr 02 '14 at 13:12
  • Ok, regarding the second step: Imagine that you see the webcam image on your screen. Now you draw some circles at locations (0, 0), (camImage.width, 0), (camImage.width, camImage.height) and (0, camImage.height). Those circles will get projected somewhere on the desk, but you will also see them on the webcam image at locations (x0, y0), (x1, y1), (x2, y2), (x3, y3). I am learning a homography that maps (0, 0) to (x0, y0), (camImage.width, 0) to (x1, y1), (camImage.width, camImage.height) to (x2, y2) etc. – Mihai Morariu Apr 02 '14 at 13:27
  • @Mihai Morariu are your page corners detected correctly (you can drawn them on the image)? is your homography recalculated when rotating the page? does your current method work if you rotate the books top/bottom instead of the page? – Micka Apr 02 '14 at 13:31
  • Regarding step 4: let's say that you see the corners of the page at locations (x'0, y'0), (x'1, y'1), (x'2, y'2), (x'3, y'3) in the webcam image. I apply the inverse of the transformation that I computed at step 2 to those points, to get a new set of points in the webcam image, at locations (x''0, y''0), (x''1, y''1), (x''2, y''2), (x''3, y''3). I warp the image that I want to project to this set of points and draw it on top of the webcam image. – Mihai Morariu Apr 02 '14 at 13:32
  • @Micka: - The page corners are detected correctly (I checked that by drawing them onto the webcam image). - The homography is not recalculated when rotating the page. - The current method works well if I translate/rotate the notebook in plane (on the desk), but not if you want to tilt it. – Mihai Morariu Apr 02 '14 at 13:33
  • I guess you have to recompute all the time. The image really looks like the projection assumes the page to be still lying plane on the desk. How do you achieve that in-plane invariance without recomputation of the homography?? – Micka Apr 02 '14 at 13:36
  • @MihaiMorariu You are mixing up 'webcam image' and 'projector image', but that's ok I get it now. So as Micka said, unless the notebook remains in the same plane, you need to recompute the homography each time you move it. – BConic Apr 02 '14 at 13:41
  • @Micka: The only thing that I can use to "see" is the webcam. What I have right now is a homography that tells me: if I have these points in the webcam image, then after projecting them on the desk these will be their new locations (also in the webcam image). I only do that once and I calibrate with the four corner points of the webcam image. Of course, maybe what I am doing is wrong. So what homography should I compute at every frame? – Mihai Morariu Apr 02 '14 at 13:47
  • @MihaiMorariu actually you have two images: the 'webcam image' and the 'projected image'. From what you say, it seems that you are projecting the webcam image, but you could project a completely different one. Anyway, the homography that you learn in step 2 is only valid as long as the projection plane (i.e. the notebook page) is constant. Hence, you can translate the notebook or rotate it in this plane, but you cannot close it otherwise the projection plane changes and you need to learn a new homography. – BConic Apr 02 '14 at 13:53
  • an additional error could occur when webcam and projector positions differ too much, if your object (page) then is too far away from the desk, the projection coordinates don't fit your assumption about the rays hitting the desk (projection and webcam rays aren't identical), but the homography should be your first thing to do. – Micka Apr 02 '14 at 14:16
  • Thank you for your explanations, guys! It makes more sense now. Cheers! – Mihai Morariu Apr 02 '14 at 18:17

1 Answers1

1

You want an effect that is called a key stone correction. The problem you are experiencing is most probably due to the fact that optical axes, positions, and focal lengths of a web camera and a projector are different. I suggest to calibrate your setup so you would know their relative pose and incorporate it in your inverse Homography.

Vlad
  • 4,425
  • 1
  • 30
  • 39