Selecting a point from a point cloud

Question

Google's Project Tango provides a point cloud, i.e. a floatBuffer with the positions xyz of a set of points in meters. I would like to be able to select one of such points by touching it on the screen.

What would be the best/easiest way to achieve this?

UPDATE

I include the code so far, as suggested I tried getting the projection of points on the screen, however after displaying the points I find I get values that are too small (i.e. 0.5,0.7 etc). I'm not working with unity but with android studio and therefore I don't have the method cam.WorldToScreenPoint(m_points[it]), I do however have a projection matrix, but I guess this is incorrect (maybe because we should go from meters to pixels). What would be the correct matrix to achieve this?

private void selectClosestCloundPoint(float x, float y) {
    //Get the current rotation matrix
    Matrix4 projMatrix =  mRenderer.getCurrentCamera().getProjectionMatrix();



    //Get all the points in the pointcloud and store them as 3D points
    FloatBuffer pointsBuffer =  mPointCloudManager.updateAndGetLatestPointCloudRenderBuffer().floatBuffer;
    Vector3[] points3D = new Vector3[pointsBuffer.capacity()/3];

    int j =0;
    for (int i = 0; i < pointsBuffer.capacity() - 3; i = i + 3) {

        points3D[j]= new Vector3(
                pointsBuffer.get(i),
                pointsBuffer.get(i+1),
                pointsBuffer.get(i+2));

        j++;
    }


    //Get the projection of the points in the screen.
    Vector3[] points2D = new Vector3[points3D.length];
    for(int i =0; i < points3D.length-1;i++)
    {
        Log.v("Points", "X: " +points3D[i].x + "\tY: "+ points3D[i].y +"\tZ: "+ points3D[i].z );
        points2D[i] = points3D[i].multiply(projMatrix);
        Log.v("Points", "pX: " +points2D[i].x + "\tpY: "+ points2D[i].y +"\tpZ: "+ points2D[i].z );
    }
}

I use a vector3 since that's the return type, but as I understand, the third component of it is not important.

nbsrujan · Answer 1 · 2015-12-23T12:08:37.540

2

Transform all points of 3D pointcloud onto image plane using camera intrinsic. Find distance between all projected points on image plane to touch coordinate on screen. Select 3D point corresponding to minimum distance or within a threshold from screen coordinate. Code snippet is given below

   for (int it = 0; it < m_pointsCount; ++it)
    {
        Vector3 screenPos3 = cam.WorldToScreenPoint(m_points[it]);
        Vector2 screenPos = new Vector2(screenPos3.x, screenPos3.y);
        float distSqr = Vector2.SqrMagnitude(screenPos - touchPos);
        if (distSqr > sqMaxDist)
        {
            continue;
        }
        closePoints.Add(it);
    }

Method I am suggesting might be computationally expensive though.

edited Dec 23 '15 at 12:08

answered Dec 23 '15 at 11:54

nbsrujan

1,179
1
12
26

Hi, thanks for your reply, and sorry for not answering sooner. what you say about the camera intrinsic sounds correct, however I don't have a world toScreenPoint method, how can I compute a projection matrix that achieves the same thing? The one I found in the Camera seems to be giving wrong values – Girauder Jan 02 '16 at 12:08
@Girauder I have calibrated RGB camera, and found intrinsics of RGB Camera. Then Used same matrix to project 3D points on camera plane. This worked pretty well. Project-Tango java api is about to provide a functionality which can be used directly in your method. https://developers.google.com/project-tango/apis/java/reference/TangoXyzIjData#ijParcelFileDescriptor – nbsrujan Jan 04 '16 at 06:54
do you mean the ij in TangoXYZij ? I thought that was more like the index and not a position on the screen. The projection matrix I'm using should already have camera intrinsics, is multiplying it by the vector3 point the right way to project the point on the screen? or am I missing something? – Girauder Jan 04 '16 at 09:03
@Girauder Yes, but this functionality is not available yet. – nbsrujan Jan 04 '16 at 09:04

Selecting a point from a point cloud

1 Answers1