UnProjected mouse coordinates are between 0-1

Question

I'm trying to create a ray from my mouse location out into 3D space, and apparently in order to do that I need to "UnProject()" it.

Doing so will give me a value between 0 & 1 for each axis.

This can't be right for drawing a "Ray" or a line from the viewport, can it? All this is, is a percentage essentially of my mouse to viewport size.

If this is actually right, then I don't understand the following:

I draw triangles that have vertices that are not constrained from 0-1, rather they are coordinates like (0,100,0), (100,100,0), (100,0,0), And these draw perfectly fine
But also, drawing the vertices that are unprojected from my mouse coordinates as lines/points also draw perfectly fine.
How the heck would I then compare my mouse coordinates to the coordinates of my objects?

If this is actually wrong, then what can cause such an error? I tried unprojecting my own object's vertices, and those aren't constrained from 0-1. I don't know whether or not the way I handle my "projections" when rendering is even compatible with gluUnproject. I've been just doing it the way these tutorials here show it (near bottom): http://qt-project.org/wiki/Developer-Guides#28810c65dd0f273a567b83a48839d275

This is the way I try to get my mouse coordinates:

GLdouble modelViewMatrix[16];
GLdouble projectionMatrix[16];
GLint viewport[4];
GLfloat winX, winY, winZ;
glGetDoublev(GL_MODELVIEW_MATRIX, modelViewMatrix);
glGetDoublev(GL_PROJECTION_MATRIX, projectionMatrix);
glGetIntegerv(GL_VIEWPORT, viewport);

winX = (float)x;
winY = (float)viewport[3] - (float)y;
glReadPixels( winX, winY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &winZ );

GLdouble nearPlaneLocation[3];
gluUnProject(winX, winY, 0, modelViewMatrix, projectionMatrix,
             viewport, &nearPlaneLocation[0], &nearPlaneLocation[1],
        &nearPlaneLocation[2]);

GLdouble farPlaneLocation[3];
gluUnProject(winX, winY, 1, modelViewMatrix, projectionMatrix,
             viewport, &farPlaneLocation[0], &farPlaneLocation[1],
        &farPlaneLocation[2]);

QVector3D nearP = QVector3D(nearPlaneLocation[0], nearPlaneLocation[1],
        nearPlaneLocation[2]);
QVector3D farP = QVector3D(farPlaneLocation[0], farPlaneLocation[1],
        farPlaneLocation[2]);

Perhaps my actual projections are off?

void oglWidget::paintGL()
{
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
    QMatrix4x4 mMatrix;
    QMatrix4x4 vMatrix;

    QMatrix4x4 cameraTransformation;
    cameraTransformation.rotate(alpha, 0, 1, 0);
    cameraTransformation.rotate(beta, 1, 0, 0);

    QVector3D cameraPosition = cameraTransformation * QVector3D(camX, camY, distance);
    QVector3D cameraUpDirection = cameraTransformation * QVector3D(0, 1, 0);
    vMatrix.lookAt(cameraPosition, QVector3D(camX, camY, 0), cameraUpDirection);
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();
    gluLookAt(cameraPosition.x(), cameraPosition.y(), cameraPosition.z(), camX, camY, 0, cameraUpDirection.x(), cameraUpDirection.y(), cameraUpDirection.z());
    shaderProgram.bind();
    shaderProgram.setUniformValue("mvpMatrix", pMatrix * vMatrix * mMatrix);
    shaderProgram.setUniformValue("texture", 0);


    for (int x = 0; x < tileCount; x++)
    {
        shaderProgram.setAttributeArray("vertex", tiles[x]->vertices.constData());
        shaderProgram.enableAttributeArray("vertex");
        shaderProgram.setAttributeArray("textureCoordinate", textureCoordinates.constData());
        shaderProgram.enableAttributeArray("textureCoordinate");

        //Triangle Drawing
        glTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, tiles[x]->image.width(), tiles[x]->image.height(), 0, GL_RGBA, GL_UNSIGNED_BYTE, tiles[x]->image.bits());
        glDrawArrays(GL_TRIANGLES, 0, tiles[x]->vertices.size());
    }    

    shaderProgram.release();

}

Where as, pMatrix is a 4x4 matrix, controlled during resize events like:

 pMatrix.setToIdentity();
 pMatrix.perspective(fov, (float) width / (float) height, 0.001, 10000);
 glViewport(0, 0, width, height);

and my vertex shader is set up like this:

uniform mat4 mvpMatrix;

in vec4 vertex;
in vec2 textureCoordinate;

out vec2 varyingTextureCoordinate;

void main(void)
{
    varyingTextureCoordinate = textureCoordinate;
    gl_Position = mvpMatrix * vertex;
}

What are you using this ray for? You already read the depth buffer back (this stores window-space Z), so you can UnProject your mouse coordinate and get an exact position in object-space. — Andon M. Coleman, Feb 11 '15 at 21:36
line-plane/triangle intersection. My ray's points only seem to lie on screen space, as the returned values are between 0-1 (like 0.003x, -0.5y, 0z). My objects on the other hand have vertices that aren't limited like this. — Yattabyte, Feb 11 '15 at 21:41
Coordinates in the range [0-1] don't really describe screen space (this is a coordinate space where 1 unit is equal in size to 1 pixel). It is not often that you draw into a 1x1 viewport ;) Those coordinates are more along the lines of Normalized Device Coordinates (which has a range [-1,1]). This suggests to me that you are not properly applying the viewport transform, I do not know why though. — Andon M. Coleman, Feb 11 '15 at 21:45
@AndonM.Coleman I have a hunch that the method that I control my "viewport" within my render function may be antiquated or obsolete, and not being used appropriately by functions like gluUnproject().Would you be able to take a quick look at my render functions? This is such a stubborn bug and I would really like to understand the reason behind all of this suffering — Yattabyte, Feb 11 '15 at 21:51
I looked through your code and the only unusual thing I see is that you are duplicating the `LookAt` work in your matrix class and using GLU, but the perspective projection matrix is only setup in `pMatrix`. What this means is that when you query the projection matrix from GL it is probably going to be an identity matrix. Instead of querying the matrices, have you considered using `vMatrix * mMatrix` and `pMatrix`? I think you might need to compensate for `float` vs. `double`, but this is the more practical approach. — Andon M. Coleman, Feb 12 '15 at 23:17

score 0 · Answer 1 · answered Feb 12 '15 at 05:19

0

glReadPixels takes integers (x and y) and you don't seem to be using winZ for some reason in gluUnProject.

Try it like this:

gluUnProject(winX, winY, winZ, glView, glProjection, viewport, &posX, &posY, &posZ);

Also, if you want the ray to stop when it meets something in the depth buffer then don't clear the depth buffer after rendering. If you do a glClear(GL_DEPTH_BUFFER_BIT) then the ray should go as far as the far clip you set in your projection matrix.

I also have no idea why you need to call it more than once. The last three floats will be the target vector and you can just use your camera position as the source of the ray (depending on what you are doing).

answered Feb 12 '15 at 05:19

Rebirth

478
1
4
13

the 2 depths (and 2 unprojections) are because I'm supposed to (as the other methods have depicted) get 2 points in space at the near and far planes of the projection, in order to draw a line between them which would then be used for line-plane/triangle intersection ("Ray-picking) – Yattabyte Feb 12 '15 at 05:24
But you aren't using the result (float) from glReadPixels. – Rebirth Feb 12 '15 at 05:37

score 0 · Accepted Answer · edited Jan 18 '21 at 12:35

Part of my problem here was poorly describing it. I accidentally left residual code from frantically testing, resulting in bits of "read Pixel" functions and related nonsense which wasn't useful for solving the problem.

The rest of my problem was due to inconsistent data types for the matrices, and trying to pull matrices from OpenGL when it never had them stored in the first place.

The problem was solved by:

Using GLM to hold all my matrices
performing the calculations myself (inverse view matrix * inverse model matrix * inverse projection matrix) * vector holding NDC converted screen space coordinates (range of -1 to 1: x or y divided by width or height, * 2 - 1), which also has a z of -1 or 1 for the far or near planes, and a w of 1.
Divide result by the fourth spot of the vector.

I still do not know why unprojecting doesn't work for me, as I got the wrong results with GLU as well as GLM's unproject function, but doing it manually worked for me.

Since my problem extended over quite a great length of time, and took up several questions, I owe credit to a few individuals who helped me along the way:

srobins of facepunch, in this thread
derhass from here, in this question, and this discussion

UnProjected mouse coordinates are between 0-1

2 Answers2

Linked