Handling touch events in a 3D "scene" or Screen to 3D coordinates

Question

I'm trying to implement a "whack-a-mole" type game using 3D (OpenGL ES) in Android. For now, I have ONE 3D shape (spinning cube) at the screen at any given time that represents my "mole". I have a touch event handler in my view which randomly sets some x,y values in my renderer, causing the cube to move around (using glTranslatef()).

I've yet to come across any tutorial or documentation that completely bridges the screen touch events to a 3D scene. I've done a lot of legwork to get to where I'm at but I can't seem to figure this out the rest of the way.

From developer.andrdoid.com I'm using what I guess could be considered helper classes for the Matrices: MatrixGrabber.java, MatrixStack.java and MatrixTrackingGL.java.

I use those classes in my GLU.glUnProject method which is supposed to do the conversion from the real screen coordinates to the 3D or object coordinates.

Snippet:

    MatrixGrabber mg = new MatrixGrabber();
    int viewport[] = {0, 0, renderer._width, renderer._height};
    mg.getCurrentModelView(renderer.myg);
    mg.getCurrentProjection(renderer.myg);
    float nearCoords[] = { 0.0f, 0.0f, 0.0f, 0.0f };
    float farCoords[] = { 0.0f, 0.0f, 0.0f, 0.0f };
    float x = event.getX();
    float y = event.getY();
    GLU.gluUnProject(x, y, -1.0f, mg.mModelView, 0, mg.mProjection , 0, viewport, 0, nearCoords, 0)
    GLU.gluUnProject(x, y, 1.0f, mg.mModelView, 0, mg.mProjection , 0, viewport, 0, farCoords, 0)

This snippet executes without error put the output does not look correct. I know the screen has the origin (0,0) at the bottom left. And the 3D scene, at least mine, seems to have the origin right at the middle of the screen like a classic cartesian system. So run my code where the screen coordinates are (0, 718) from touching the bottom left. My outputs from last parameters to gluUnProject are:

Near: {-2.544, 2.927, 2.839, 1.99}

Far: {0.083, 0.802, -0.760, 0.009}

Those numbers don't make any sense to me. My touch even was in the 3rd quadrant so all my x,y values for near and far should be negative but they aren't. The gluUnProject documention doesn't mention any need to convert the screen coordinates. Then again, that same documentation would lead you to believe that Near and Far should have been arrays of size 3 but they have to be of size 4 and I have NO CLUE why.

So, I've got two questions (I'm sure more will come up).

1. How can I make sure I am getting the proper near and far coordinates based on the screen coordinates.
2. Once I have the near and far coordinates, how do I use that to find if the line they create intersects an object on the screen.

Answer 1

I remember runnning into problems with glUnProject on Android back in my college days. (That was in the early days of Android) One of my fellow students figured out that our calculations would get mangled by the 4th dimension in the result of glUnProject. If I recall correctly, this was something documented somewhere, but for some reason I haven't been able to dig that up again. I never dug into the specifics of it, but perhaps what helped us may also be of use to you. It's likely to do with the math we applied...

/**
 * Convert the 4D input into 3D space (or something like that, otherwise the gluUnproject values are incorrect)
 * @param v 4D input
 * @return 3D output
 */
private static float[] fixW(float[] v) { 
    float w = v[3];
    for(int i = 0; i < 4; i++) 
        v[i] = v[i] / w;
    return v;
}

We actually used the above method to fix up the glUnProject results and do a pick/touch/select action on spherical objects in 3D space. Below code may provide a guide on how to do this. It's little more than casting a ray and doing a ray-sphere intersection test.

A few additional notes that may make below code more easy to understand:

• Vector3f is a custom implementation of a 3D vector based on 3 float values and implements the usual vector operations.
• shootTarget is the spherical object in 3D space.
• The 0 in calls like getXZBoundsInWorldspace(0) and getPosition(0) are simply an index. We implemented 3D model animations and the index determines which 'frame/pose' of the model to return. Since we ended up doing this specific hit test on a non-animated object, we always used the first frame.
• Concepts.w and Concepts.h are simply the width and height of the screen in pixels - or perhaps differently said for a full screen app: the screen's resolution.

_

/**
 * Checks if the ray, casted from the pixel touched on-screen, hits
 * the shoot target (a sphere). 
 * @param x
 * @param y
 * @return Whether the target is hit
 */
public static boolean rayHitsTarget(float x, float y) {
    float[] bounds = Level.shootTarget.getXZBoundsInWorldspace(0);
    float radius = (bounds[1] - bounds[0]) / 2f;
    Ray ray = shootRay(x, y);
    float a = ray.direction.dot(ray.direction);  // = 1
    float b = ray.direction.mul(2).dot(ray.near.min(Level.shootTarget.getPosition(0)));
    float c = (ray.near.min(Level.shootTarget.getPosition(0))).dot(ray.near.min(Level.shootTarget.getPosition(0))) - (radius*radius);

    return (((b * b) - (4 * a * c)) >= 0 );

}

/**
 * Casts a ray from screen coordinates x and y.
 * @param x
 * @param y
 * @return Ray fired from screen coordinate (x,y)
 */
public static Ray shootRay(float x, float y){
    float[] resultNear = {0,0,0,1};
    float[] resultFar = {0,0,0,1};

    float[] modelViewMatrix = new float[16];
    Render.viewStack.getMatrix(modelViewMatrix, 0);

    float[] projectionMatrix = new float[16];
    Render.projectionStack.getMatrix(projectionMatrix, 0);

    int[] viewport = { 0, 0, Concepts.w, Concepts.h };

    float x1 = x;
    float y1 = viewport[3] - y;

    GLU.gluUnProject(x1, y1, 0.01f, modelViewMatrix, 0, projectionMatrix, 0, viewport, 0, resultNear, 0);
    GLU.gluUnProject(x1, y1, 50f, modelViewMatrix, 0, projectionMatrix, 0, viewport, 0, resultFar, 0);
    //transform the results from 4d to 3d coordinates.
    resultNear = fixW(resultNear);
    resultFar = fixW(resultFar);
    //create the vector of the ray.
    Vector3f rayDirection = new Vector3f(resultFar[0]-resultNear[0], resultFar[1]-resultNear[1], resultFar[2]-resultNear[2]);
    //normalize the ray.
    rayDirection = rayDirection.normalize();
    return new Ray(rayDirection, resultNear, resultFar);
}

/**
 * @author MH
 * Provides some accessors for a casted ray.
 */
public static class Ray {
    Vector3f direction;
    Vector3f near;
    Vector3f far;

    /**
     * Casts a new ray based on the given direction, near and far params. 
     * @param direction
     * @param near
     * @param far
     */
    public Ray(Vector3f direction, float[] near, float[] far){
        this.direction = direction;
        this.near = new Vector3f(near[0], near[1], near[2]);
        this.far = new Vector3f(far[0], far[1], far[2]);
    }
}

Answer 2

How can I make sure I am getting the proper near and far coordinates based on the screen coordinates.

To begin with, read this answer if you haven't read information about GLU.glProject somewhere else already. GLU.glUnProject does the exact inverse of that function, which I found more easy to understand, and helps to understand the concept of mapping screen coordinates into object space. Worked for me at least.

If you want to test you're getting correct values from GLU.glUnProject it's easiest done if you start with some easy to understand projection and model-view -matrices. Here's a code snippet I was using earlier today;

  // Orthographic projection for the sake of simplicity.
  float projM[] = new float[16];
  Matrix.orthoM(projM, 0, -ratio, ratio, 1f, -1f, zNear, zFar);

  // Model-View matrix is simply a 180 degree rotation around z -axis.
  float mvM[] = new float[16];
  Matrix.setLookAtM(mvM, 0, 0f, 0f, eyeZ, 0f, 0f, 0f, 0f, 1f, 0f);
  Matrix.rotateM(mvM, 0, 180f, 0f, 0f, 1f);

  // (0, 0) is top left point of screen.
  float x = (width - (width / ratio)) / 2;
  float y = height;
  float objNear[] = new float[4];
  float objFar[] = new float[4];
  int view[] = { 0, 0 ,width, height };
  // --> objNear = { 1, 1, eyeZ - zNear  }
  GLU.gluUnProject(x, y, 0, mvM, 0, projM, 0, view, 0, objNear, 0);
  // --> objFar = { 1, 1, eyeZ - zFar }
  GLU.gluUnProject(x, y, 1, mvM, 0, projM, 0, view, 0, objFar, 0);

With more complex model-view and projection matrices it becomes rather difficult to verify you're getting coordinates in object space you're expecting. And it might be a good starting point to play around with some easy to understand matrices first. Once you're happy with the results you shouldn't have to worry about GLU.glUnProject at all.

Once I have the near and far coordinates, how do I use that to find if the line they create intersects an object on the screen.

For hit testing in 3d object space, it should be easiest to pre-compute bounding sphere or bounding box for your objects. Once you want to check if user clicked on an object, and have two screen points unProjected into object space, you have two points on a line in object space.

If you're using bounding sphere you can calculate intersection points with this line, or alternatively, only sphere center point distance. In former case should there be at least one intersection and in latter one distance should be less than sphere radius.

For bounding box this question is a good read.