In the case of a well-defined symmetric view frustum, the solution can be found with some simple geometry:

I assume YY
is in normalized device coordinates (ranging from -1 to 1). If they are in pixel coordinates, you have to convert them.
I have chosen the projection plane to be 1 unit far away from the camera. But any other distance would work as well. Then, the distance y'
is simply
y' = YY * H/2
H/2
is the half screen height and can be calculated with:
H/2 = tan (fovy/2)
where fovy
is the camera's field of view in the vertical direction.
You want to find angle alpha
for your rotation. This is simply:
tan alpha = y' / 1 = YY * tan(fovy / 2)
alpha = atan(YY * tan(fovy / 2)
Be aware of the direction. Positive values specify downward rotations.
For arbitrary projections, this problem can be solved analytically:
Assuming we have projection matrix P
and view matrix V
, we want to solve for:
w-clip(P * V * (0 0 1 0)^T) = (... YY ...)
Since you only want to allow translations and rotations about the x-axis for your camera, V
has the form:
/ 1 0 0 tx \
V = | 0 cos alpha -sin alpha ty |
| 0 sin alpha cos alpha tz |
\ 0 0 0 1 /
This yields the equation:
YY = (p23 * cos alpha - p22 * sin alpha) / (p43 * cos alpha - p42 * sin alpha)
where pij
is the entry of P
in the i
-th row and j
-th column.
Use your favorite symbolic solver to get a solution for alpha
and you can re-calculate your view-matrix.