Using gluUnProject to map touches to x,y cords on z=0 plane in Android OpenGL ES 2.0

Question

I've drawn a grid at z=0 in OpenGL 2.0 ES, and just want to convert touch inputs to x/y coordinates on that plane. It seems like this is best done through ray tracing, which involves running gluUnProject on 0 and 1, then creating a ray, solving that ray for z=0?

I found this code, but it is OpenGL ES 1.0: i-schuetz / Android_OpenGL_Picking

Screenshot of app running so you can see camera distortion.

My code on Github, only 4 files. The unproject function I'm trying to write is in MyGLRenderer.java:

    public float[] unproject(float rx, float ry) {
            float rz = 1;
        float[] xyzw = {0, 0, 0, 0};
        int[] viewport = {0, 0, mDisplayWidth, mDisplayHeight};
        android.opengl.GLU.gluUnProject(
                        rx, ry, rz, // window coordinates
                        mViewMatrix, 0,
                        mProjectionMatrix, 0, 
                        viewport, 0,
                        xyzw, 0);
        xyzw[0] /= xyzw[3];
        xyzw[1] /= xyzw[3];
        xyzw[2] /= xyzw[3];
        xyzw[3] = 1;
        return xyzw;
    }

I would like this function to take an rx and ry for the screen, and return an rx and ry for the z=0 plane.

Andon M. Coleman · Accepted Answer · 2013-12-20T02:03:37.993

2

There is nothing particularly special about what gluUnProject (...) does. If you have all of the matrices and the viewport dimensions _{(x,y and w_idth,h_eight)} I can walk you through the process of implementing it yourself.

_{NOTE: I tend to call each coordinate space by a different name than you might be used to, understand that screen space is another name for window space, view space is another name for eye space, object space is another name for model space.}

Step 1:

Screen Space to NDC space _{(Undo Viewport Transform)}

NDC_X = (2.0 × (Screen_X - Viewport_X) / Viewport_W) - 1.0

NDC_Y = (2.0 × (Screen_Y - Viewport_Y) / Viewport_H) - 1.0

Screen Space to NDC space _{(Undo Depth Range Mapping)}

Typically in screen space, the Depth Range will map z=0 to near and z=1 to far:

NDC_Z = 2.0 × Screen_Z - 1.0

Step 2^{^†}:

NDC space to Object space _{(Undo Projection, View and Model Transforms)}

(Projection Matrix)^-1 × NDC_XYZ1 = View_XYZW

(ModelView Matrix)^-1 × View_XYZW = Object_XYZW

†_{This can actually be combined into a single step, as you will see below...}

Object_XYZw = (Projection Matrix × ModelView Matrix)^-1 × NDC_XYZ1

Now, you may notice that I crossed-out W in Object_XYZ, we really do not care about this at all but the math will produce a pesky W value nevertheless. At this point, you can return the individual components of Object_XYZ as your r_X, r_Y and r_Z.

edited Dec 20 '13 at 02:03

answered Dec 20 '13 at 01:38

Andon M. Coleman

42,359
2
81
106

Thank you! I'm coding up your answer now. – Robert Kuykendall Dec 20 '13 at 17:54
So I finished implementation, but it seems like I'm exactly where I started. I have a single XYZ point, but it's not on the Z=0 plane. I think I might need to use that point, and the camera point, and solve them for z? Here is where I coded up your answer: https://github.com/rkuykendall/ProjectRome/commit/36a5b8073461e9e0d1593daf38b0bd8dfd189620 – Robert Kuykendall Dec 21 '13 at 01:37
@RobertKuykendall: Also, if you are trying to use this for picking you have only solved half of the equation by unprotecting a single point. What you would want to do is solve (find the equivalent object space position) for a point at screen space (x,y,0) and a point at screen space (x,y,1) then compute a ray that passes through both points (and starts at the first point you solved for). Once you have this ray, you would use it to perform intersection tests vs. anything in your scene. – Andon M. Coleman Dec 21 '13 at 02:13
I think the "near" and "far" clipping planes are in reference to the camera, and not the drawing plane. I'm referring to the plane the [items are drawn on](http://i.imgur.com/hVyoN5X.png). I drew all those tiles at z=0, and now I'm trying to find out which one was clicked. I don't know how the xyz cord I have will help me find where on the drawing surface I clicked, since I drew things at z=0. – Robert Kuykendall Dec 21 '13 at 02:16
Obviously I can work out the math of which tile I clicked if I have an XY cord on the Z=0 plane where I drew them, but I can't figure out how to get that point. – Robert Kuykendall Dec 21 '13 at 02:18
Thank you. Sorry, I wrote my last 2 messages before refreshing and seeing your message. I will work on creating a ray and tracing it to z=0. – Robert Kuykendall Dec 21 '13 at 02:20
Have a look at [this article](http://msdn.microsoft.com/en-us/library/windows/desktop/bb206341(v=vs.85).aspx). The analogy used in the second paragraph of **The Viewing Frustum** may help you tremendously. In screen space, z=0 is the closest possible point and z=1 is the farthest (depth range can alter this). By shooting a ray from ScreenZ=0 to ScreenZ=1, you then have a ray that covers all possible points that project to: (x,y,[0,1]). Anything beyond z=[0,1] lies outside the viewing volume and can be ignored, but rays are nicer for intersection tests than line segments so just use a ray =P – Andon M. Coleman Dec 21 '13 at 02:39

Using gluUnProject to map touches to x,y cords on z=0 plane in Android OpenGL ES 2.0

1 Answers1

Step 1:

Screen Space to NDC space _{(Undo Viewport Transform)}

Screen Space to NDC space _{(Undo Depth Range Mapping)}

Step 2^{^†}:

NDC space to Object space _{(Undo Projection, View and Model Transforms)}

Linked

Using gluUnProject to map touches to x,y cords on z=0 plane in Android OpenGL ES 2.0

1 Answers1

Step 1:

Screen Space to NDC space (Undo Viewport Transform)

Screen Space to NDC space (Undo Depth Range Mapping)

Step 2†:

NDC space to Object space (Undo Projection, View and Model Transforms)

Linked

Screen Space to NDC space _{(Undo Viewport Transform)}

Screen Space to NDC space _{(Undo Depth Range Mapping)}

Step 2^{^†}:

NDC space to Object space _{(Undo Projection, View and Model Transforms)}