Maya C++ plugin needing to frame object with viewport from quarter camera perspective

Question

After countless hours of attempting to solve and research this issue on my own I've come up empty coming ever so close to a perfect fit in the Maya viewport to no avail. I've had attempts get extremely close and others leave me on the floor. At this point I'm lost since there is next to no info online that I could find about using the various matrices that exist and how to piece them together to solve my problem.

This is my closest attempt I've gotten and I know there is just one or two issues wrong here I just can't figure out just what it is. I'm attempting to use a projection view model matrix but I can't be sure the right way to get it or if an inclusive matrix is the correct one to use. I've been using this answer here which has been moderately useful at at least steering me in the right direction. It uses a very inefficient means of getting the two farthest points it can and bonus points if you can help me with that. In conclusion I am almost certain the issue has something to do with the matrix I'm using or the MVector I'm getting from viewDirection. Thanks in advance!

MStatus FrameCameras::positionCamera(double aspect, double fov)
{
    MDagPath cur_obj;
    MPointArray points;
    MSelectionList united_obj;
    MStatus cur_status;
    vector<MPoint> obj_points;
        
    MDagPath cur_cam_path;
    m_CameraList.getDagPath(0, cur_cam_path);
    MFnCamera cam_fn(cur_cam_path);
    MFnTransform cam_trans(cur_cam_path);
    MFloatMatrix cam_proj_matrix = cam_fn.projectionMatrix();
    
    MGlobal::executeCommand("flushUndo;");
    MGlobal::executeCommand("undoInfo -st true -l 1;");
    MGlobal::setActiveSelectionList(m_OriginalSelection);
    MGlobal::executeCommand("polyUnite -cch false -muv 0;", false, true);
    
    MGlobal::getActiveSelectionList(united_obj);
    united_obj.getDagPath(0, cur_obj);
    MFnMesh mesh(cur_obj);
    
    mesh.getPoints(points, MSpace::kWorld);
    if (points.length() <= 0)
    {
        MGlobal::displayError("Could not get points from mesh " + MString(mesh.name()));
        MGlobal::executeCommand("undo;", false, true);
        MGlobal::executeCommand("undoInfo -st false;");
        return MS::kFailure;
    }
    
    MGlobal::executeCommand("undo;", false, true);
    MGlobal::executeCommand("undoInfo -st false;");
    
    cur_status = M3dView::getM3dViewFromModelPanel(MString(m_cameraPanels["quarter_cam"].c_str()), cur_viewport);
    if (cur_status != MS::kSuccess)
    {
        MGlobal::displayError("Could not get viewport for quarter_cam:" + cur_status.errorString());
        return MS::kFailure;
    }
    
    MMatrix proj_model_view_matrix = cur_cam_path.inclusiveMatrix();
    
    for (MPoint pos : points)
    {
        MPoint val = proj_model_view_matrix * MPoint(pos.x, pos.y, pos.z, 1);
        obj_points.push_back(val);
        //obj_points.push_back(pos);
    }
    
    MPoint A1, B1, A2, B2;
    
    double width = 0;
    double height = 0;
    
    for (int i = 0; i < mesh.numVertices() - 1; i++)
    {
        for (int j = i; j < mesh.numVertices(); j++)
        {
            double new_width = abs(obj_points[i].x - obj_points[j].x);
            double new_height = abs(obj_points[i].y - obj_points[j].y);
    
            if (new_width > width)
            {
                A1 = obj_points[i];
                B1 = obj_points[j];
                width = new_width;
            }
            if (new_height > height)
            {
                A2 = obj_points[i];
                B2 = obj_points[j];
                height = new_height;
            }
        }
    }
    
    MPoint A, B;
    
    double dist;
    if (height * aspect > width)
    {
        dist = height / 2.0 / tan(fov / 2.0);
        A = A1;
        B = B1;
    }
    else
    {
        dist = width / 2.0 / tan(fov * aspect / 2.0);
        A = A2;
        B = B2;
    }
    
    MPoint center = MPoint((A + B) / 2.0);
    MVector camera_direction = cam_fn.viewDirection(MSpace::kWorld);
    camera_direction.normalize();
    MPoint camera_pos = center - camera_direction * dist;
    
    cam_trans.setTranslation(camera_pos, MSpace::kWorld);
    
    return MS::kSuccess;
}

Additionally here's the quarter_cam's properties here if that helps with anything.

EDIT: I finally got it down to a proper projection view model matrix that should be getting the points to screen space now that I've additionally adjusted the function to gather points from object space rather than world space to avoid conflict. The only issue now is that it still doesn't work.

// Gather projection and modelview matrices
MMatrix model = obj_trans.transformationMatrix();
MMatrix view = cam_fn.transformationMatrix().inverse();
MFloatMatrix f_proj = cam_fn.projectionMatrix();
MMatrix proj = f_proj.matrix;
MMatrix proj_view_model = proj * view * model;
MMatrix model_view;
cur_viewport.modelViewMatrix(model_view);

Here is what I currently use now to get the projected model view matrix however both view and model are exactly the same being:

{{1, 0, 0, 0},
{0, 1, 0, 0},
{0, 0, 1, 0},
{0, 0, 0, 1}}

In turn the resulting projection model view just becomes the projection matrix.

When it comes to the modelViewMatrix function from the viewport it returns this:

{{-0.7071, -0.3368, 0.6217, -0.0},
{5.5511e-17, 0.8792, 0.4763, -0.0},
{-0.7071, 0.3368, -0.6217, -0.0},
{5.3184, -11.5741, -480.0625, 1.0}}

which is vastly different than the previous model and view matrix would've multiplied out to be. Additionally, from what I've found, Maya uses right-hand coordinates and is row major so what I have should be correct. From all the attempts I've made w has always managed to be less than 1 suggesting that by the time I do projection division, the values are far from being within -1 to 1. I'm really hoping there's a simple fix to this but only time will tell.

Maya C++ plugin needing to frame object with viewport from quarter camera perspective

0 Answers0