0

EDIT: Solved the issue, see my own answer

Recently I've been working on a 3D world editor that will be using picking to raise or lower terrain. I'm using camera unprojection and ray casting to find the world position of the mouse screen coordinates.

However, it seems like the ray is on the wrong axis. As I remember, the ray that is supposed to come out of unproject should be coming straight from the camera.

Here's an example of what it currently looks like.

My question is, why is the ray over the Y-axis when it's supposed to be over the Z-axis?

XMFLOAT3 D3D11Camera::Unproject(const float& px, const float& py, const float& pz)
{
    const XMFLOAT2& res = D3D11RenderSettings::Instance()->resolution();

    XMVECTOR coords = XMVector3Unproject(XMVectorSet(px, res.y - py, pz, 0.0f), 0.0f, 0.0f, res.x, res.y, near_plane_, far_plane_, projection_, view_, XMMatrixIdentity());

    XMFLOAT3 to_ret;
    XMStoreFloat3(&to_ret, coords);

    return to_ret;
}

This is the unprojection code..

And this is how I'm using it

projectRay: function()
{
    var p = Mouse.position(MousePosition.Relative);
    p.x = (p.x + RenderSettings.resolution().w / 2);
    p.y = (p.y + RenderSettings.resolution().h / 2);

    var unprojA = this._camera.unproject(p.x, p.y, 0);
    var unprojB = this._camera.unproject(p.x, p.y, 1);

    var dir = Vector3D.normalise(Vector3D.sub(unprojB, unprojA));
    var ray = Ray.construct(unprojA, dir);

    var p1 = ray.origin;
    var p2 = Vector3D.add(ray.origin, Vector3D.mul(ray.direction, 1000));

    RenderTargets.ui.drawLine(p1.x, p1.y, p1.z, 1, 0, 0, p2.x, p2.y, p2.z, 1, 0, 0);

    return ray;
}

Cheers!

Olivier Moindrot
  • 27,908
  • 11
  • 92
  • 91
user3001604
  • 53
  • 7

1 Answers1

0

Stupidity aside, I'm working with both an inverted Y-axis coordinate system and a default -- non inverted -- one. It seemed like it was a ray oriented vertically, but in fact it was a ray that was oriented in the direction of '/' while it was supposed to be oriented like '\'. Multiplying the Y component of the result by -1 solved the issue.

user3001604
  • 53
  • 7