Java OpenGL: Mouse picking in 3D Space

Question

I am trying to make some sort of 3D Editor with Java and OpenGL. And now I'm implementing the basic functions of an 3D Editor like rotating the camera around a specific Position and zooming. Next I want to do a 3D Picking to select Objects,Lines and Vertices in 3D-Space with the Mouse. I thought this is gonna to be easy because I can already select Objects when the Camera is focusing them.

Here is the example of the Selection of Objects with the Camera focus:

In the Class Camera there is this Method:

    public boolean isVecInFocus(Vec3 vec) {

    //returns the distance between camera and target

    float c = new Vec3(posX,posY,posZ).getDistanceTo(vec);

    // returns a Vector by drawing an imiginary line with the length of c and the position and rotation of the camera

    Vec3 target = getFocusedPoint(c);

    //checks if the calculated Vector is near to the target

    if(target.x > vec.x - 0.05f && target.x < vec.x + 0.05f && target.y > vec.y - 0.05f && target.y < vec.y + 0.05f && target.z > vec.z - 0.05f && target.z < vec.z + 0.05f) {

        return true;
    } else {

        return false;
    }
}

Now, I want to do the same with the Mouse input:

    //Mouse positions
    float mX = Mouse.getX();
    float mY = Mouse.getY();

    //My test Vector
    Vec3 vec = new Vec3(-5,5,-8);

    //Camera Position
    Vec3 camV = new Vec3(cam.getPosX(),cam.getPosY(),cam.getPosZ());

    //Distance from Test Vector to Camera
    float c = camV.getDistanceTo(vec);

    //Calculating of the aspect between width and height (Because fov_x and fov_y are different because of the Screen Resolution, I think)
    float aspect = (float) sb.getDisplayWidth() / (float) sb.getDisplayHeight();

    //Normal fov refers to fov_y, so here is the fov_x
    float fovx = cam.fov * aspect;

    //Changing the Rotations to calculate the target Vector with the values of the Mouse position and rotations , not the Camera
    float rotY = cam.getRotationY() + (fovx / (float) sb.getDisplayWidth()) * (mX) - (fovx / 2F);
    float rotX = cam.getRotationX() + (cam.fov / (float) sb.getDisplayHeight()) * ((float) sb.getDisplayHeight() - mY) - (cam.fov / 2F);

    //Calculating the target Vector with simple Math ...

    double xDis = c * Math.sin(Math.toRadians(rotY)) * Math.cos(Math.toRadians(rotX));
    double yDis = c * Math.sin(Math.toRadians(rotX));
    double zDis = c * Math.cos(Math.toRadians(rotY)) * Math.cos(Math.toRadians(rotX));

    float posX = (float) (camV.x + xDis);
    float posY = (float) (camV.y - yDis);
    float posZ = (float) (camV.z - zDis);

    Vec3 target = new Vec3(posX,posY,posZ);

    //Check if the target Vector and the Test Vector are the same.

If I use this Code, and point with my Mouse at the Test-Vector, the result is not right. The accuracy of the Point gets lower, the bigger the difference between Screen-middle and Mouse position is.

I think it has something to do with the OpenGL Perspective, but I'm not sure ...