Imagine that at the center of the window is 3d sphere (not ellipse) that touches edges of window. It is located in screen space, not in world space. "Screen space" means that if your window is 640x480, then sphere is located at x=320,y=240 and has radius of 240 (min(width,height)/2
), regardless of your modelview/projection matrix.
When user clicks pushes mouse button down, he/she selects point on the sphere. If click is outside of that sphere, initial point moved into sphere boundary. Let's call this point pointA
. When user is dragging mouse around, mouse position marks where user wants that initial point to be. Let's call it pointB
.
Now for a sphere it is quite easy to calculate 3rd coordinate given 1st two, since R^2 = X^2+y^2+z^2
So you can calculate 3rd coordinate for the initial point (pointA
) and 3rd coordinate for current point (pointB
). Now using two points you can calculate two vectors from center of ther sphere (O
) to point (vectorOA = pointA - O; vectorOB = pointB - O
). Now since you have two vectors, you can use them to calculate rotation axis (using cross product = rotationAxis = normalize(vecOA x vecOB);
) and rotation angle (using acos on dot product of normalized vectors = acos(dot(normalize(vecOA), normalize(vecOB))
). Using axis and angle you can build rotation matrix in screen space, which can be then transformed into space you want and applied to object.
The technique is commonly called "trackball". It has been used in some examples in microsoft DirectX SDK, and you should be able to find it somewhere within NVidia OpenGL SDK. So if you don't understand explanation, I suggest to grab either sdk and investigate any example that allows you to rotate object with mouse.