This problem yields a system of equations. Let R_3d
be the rotation in 3d space, R_xy
the rotation in the xy
plane, and [*]_xy
be the projection of *
onto the xy
plane. Then for any point v
:
I: [R_3d v]_zx = R_zx [v]_zx
II: [R_3d v]_zy = R_zy [v]_zy
III: [R_3d v]_xy = R_xy [v]_xy
We see that every coordinate is present in two equations. Let's check the relevant equations for the x-coordinate:
a := alpha, b := beta, c := gamma
I: cos b cos c x - cos b sin c y + sin b z = sin Pzx z + cos Pzx x
III: cos b cos c x - cos b sin c y + sin b z = cos Pxy x - sin Pxy y
We see that the following relation mus hold for any v
(right hand side of both equations):
sin Pzx z + cos Pzx x = cos Pxy x - sin Pxy y
Similar equations exist for the other two coordinates. Only if these conditions are met, an exact 3d rotation can exist. If I'm not mistaken, that's only the case if Pzx=Pzy=Pxy=0
. In general, an approximate solution can be calculated. I would suggest a least-squares solution based on the following energy:
E(a, b, c) = Sum { for all v in data set } ( || [R_3d v]_zx - R_zx [v]_zx ||^2
+ || [R_3d v]_zy - R_zy [v]_zy ||^2
+ || [R_3d v]_xy - R_xy [v]_xy ||^2 )
And the optimal rotation parameters are:
{a, b, c}* = arg min {a, b, c} E(a,b,c)
This solution will minimize the distance of the two projections of corresponding points.
Unfortunately, the problem is not a linear least-squares problem which would be easy to solve. Instead, iterative methods can solve this problem (e.g. Levenberg–Marquardt). Look for an implementation of that algorithm in your programming language, plug in the energy and solve for the optimal rotation parameters.