Feeding a new cv::Mat into a vx_graph

Question

Given sample code like in this previous question (error checking deleted for brevity):

vx_image inputImage = vxCreateImage( graph, width, height, VX_DF_IMAGE_S16 );
vx_image outputImage = vxCreateImage( graph, width, height, VX_DF_IMAGE_S16 );
vx_graph graph = vxCreateGraph( context );
vx_image intermediate1 = vxCreateVirtualImage( graph, width, height, VX_DF_IMAGE_S16 );
vx_image intermediate2 = vxCreateVirtualImage( graph, width, height, VX_DF_IMAGE_S16 );
vx_image intermediate3 = vxCreateVirtualImage( graph, width, height, VX_DF_IMAGE_S16 );

vxSobel3x3Node(graph,inputImage,intermediate1,intermediate2);
vxMagnitudeNode(graph,intermediate1,intermediate2,intermediate3);
vxConvertDepthNode(graph,intermediate3,outputImage,VX_CONVERT_POLICY_WRAP,0);    

vxVerifyGraph( graph );

vxProcessGraph( graph );

And the classic OpenCV reading video loop:

cv::VideoCapture cap;
cap.open("/Testing/test.avi");
for(;;)
{
    cv::Mat frame;
    cap >> frame;
    if( frame.empty() ) break;
    // TODO: what goes here?
    vxProcessGraph( graph );
}

How do I get the frame into the inputImage so I can call vxProcessGraph() correctly? I know that:

vx_image image = nvx_cv::createVXImageFromCVMat(frame);

Is a helper for getting a vx_image, but I cannot seem to find an example of what to do next. Some NVidia documentation I read suggested:

vxAccessImagePatch(...);
// copy pixel-by-pixel??
vxCommitImagePatch(...);

However, this seems a bit too brute-force-like, so is there some nicer way (vxSetParameter, perhaps, but the documentation is very unclear) to do this?

score 2 · Answer 1 · answered Apr 25 '18 at 02:48

It is possible to create a vx_image which only references the data stored in a cv::Mat. Something using the vx_image to access data would end up actually accessing the data stored in the Mat. This approach uses only documented OpenCV and OpenVX functions. From this Intel documentation:

vx_image get_vx_image_from_Mat(vx_context graph, cv::Mat &mat) {
    vx_imagepatch_addressing_t frame_format {};
    frame_format.dim_x = mat.cols;
    frame_format.dim_y = mat.rows;
    return vxCreateImageFromHandle(graph, 
                                   VX_DF_IMAGE_S16, 
                                   &format, 
                                   &((void*)mat.data),
                                   VX_IMPORT_TYPE_HOST);
}

Presumably, the cv:Mat needs to survive at least as long as any vx_image that is created in this manner.

Ken Y-N · Accepted Answer · 2018-04-25T04:00:16.957

I managed to work out the solution, as what was confusing me about examples was the use of a helper function not in the official API.

vx_status vxAddParameterToGraphByIndex(vx_graph g, vx_node n, vx_uint32 index)
{
    vx_parameter p = vxGetParameterByIndex(n, index);
    vx_status status = vxAddParameterToGraph(g, p);
    vxReleaseParameter(&p);
    return status;
}

Now, my sample code above needs a quick modification:

vx_node sobel = vxSobel3x3Node(graph,inputImage,intermediate1,intermediate2);

Then the magic line of:

vxAddParameterToGraphByIndex(graph, sobel, 0);

The 0 here means the 0th parameter to the sobel node, excluding the graph itself.

Next, my loop becomes:

if( frame.empty() ) break;
vx_image image = nvx_cv::createVXImageFromCVMat(frame);
vxSetGraphParameterByIndex(graph, 0, (vx_reference) image);
vxProcessGraph( graph );
vxReleaseImage(&image);

Here, the 0 means the 0th parameter that we have added to the graph. If I had access the vx_parameter p above, there is the slightly less unfriendly:

vxSetParameterByIndex(p, (vx_reference) image);

This might work on its own, but I don't know if there's extra work to do to notify the vx_graph of the change.

Feeding a new cv::Mat into a vx_graph

2 Answers2