How to use experimental meshing API with Project Tango

Question

I apologize in advance for my long post.

My purpose is to create a meshing app for a Project Tango Yellowstone device to create a 3D map of building interiors. I intend to make use of the experimental meshing API added in recent versions of the tango-examples-c code.

I'm using point-cloud-jni-example (turing) as a starting point and so far have done the following:

1. Set config_experimental_enable_scene_reconstruction tango config parameter in point_cloud_app.cc (see docs)

   // Enable scene reconstruction
   ret = TangoConfig_setBool(tango_config_, 
           config_experimental_enable_scene_reconstruction", true);
       if (ret != TANGO_SUCCESS) {
           LOGE("PointCloudApp:         config_experimental_enable_scene_reconstruction() failed"
               "with error code: %d", ret);
           return ret;
   }
   

2. Added extractMesh native method in TangoJNINative.java

   // Extracts the full mesh from the scene reconstruction.
   public static native float extractMesh();
   

3. Added matching extractMesh function to the jni_interface.cc

   JNIEXPORT void JNICALL
   Java_com_projecttango_experiments_nativepointcloud_TangoJNINative_extractMesh(
   JNIEnv*, jobject) {
       app.ExtractMesh();
   }
   

4. Added ExtractMesh method in point_cloud_app.cc

   void PointCloudApp::ExtractMesh() {
   // see line 1245 of tango_client_api.h
       mesh_ptr = new TangoMesh_Experimental();
       TangoService_Experimental_extractMesh(mesh_ptr);
       mesh = *mesh_ptr;
       LOGE("PointCloudApp: num_vertices: %d", mesh.num_vertices);
   
       float float1, float2, float3;
       float1 = mesh.vertices[1][0];
       float2 = mesh.vertices[1][1];
       float3 = float1 + float2;  // these lines show I can use the vertex data
       LOGE("PointCloudApp: First vertex, x: %f", mesh.vertices[1][0]);  // this line causes app to crash; printing the vertex data seems to be the problem
   }
   

5. Added TangoMesh_Experimental declaration to point_cloud_app.h

   // see line 1131 of tango_client_api.h
   TangoMesh_Experimental* mesh_ptr;
   TangoMesh_Experimental mesh;
   

6. Added an additional button to call the extractMesh native method. (not showing this one as it is pretty straightforward)

For reference, here is the TangoMesh_Experimental Struct from the API:

    // A mesh, described by vertices and face indices, with optional per-vertex
    // normals and colors.
    typedef struct TangoMesh_Experimental {
    // Index into a three-dimensional fixed grid.
    int32_t index[3];

    // Array of vertices. Each vertex is an {x, y, z} coordinate triplet, in
    // meters.
    float (*vertices)[3];

    // Array of faces. Each face is an index triplet into the vertices array.
    uint32_t (*faces)[3];

    // Array of per-vertex normals. Each normal is a normalized {x, y, z} vector.
    float (*normals)[3];

    // Array of per-vertex colors. Each color is a 4-tuple of 8-bit {R, G, B, A}
    // values.
    uint8_t (*colors)[4];

    // Number of vertices, describing the size of the vertices array.
    uint32_t num_vertices;

    // Number of faces, describing the size of the faces array.
    uint32_t num_faces;

    // If true, each vertex will have an associated normal. In that case, the
    // size of the normals array will be equal to num_vertices. Otherwise, the
    // size of the normals array will be 0.
    bool has_normals;

    // If true, each vertex will have an associated color. In that case, the size
    // of the colors array will be equal to num_vertices. Otherwise, the size of
    // the colors array will be 0.
    bool has_colors;
    } TangoMesh_Experimental;

My current understanding of this struct is:

1. The three pointers in float (*vertices)[3]; point to the addresses at the start of three chuncks of memory for the x, y, and z coordinates for the vertices for the mesh (the same is true for normals and colors colors). A specific vertex is composed of an x, y, and z component found at a specific index in the three arrays.

2. Similarly, the uint32_t (*faces)[3] array has three pointers to the beginning of three chunks of memory, but a specific set of three elements here instead contains index numbers that indicate which three vertices (from the vertices array (each with three coordinates)) make up that face.

The current status is I am able to extract the mesh, and print some of it to the console, then crashes without errors

PointCloudApp: PointCloudApp: num_vertices: 8044 

If I omit the last line I added in point_cloud_app.cc (#4, above), the app doesn't crash. I am able to access the vertex data and do something with it, but printing it using LOGE causes a crash 9 times out of 10. Occasionally, it does print the value correctly without crashing. Could the vertex data have holes or invalid values?

I have tried returning test_float from JNI back to java, but it crashes again when I try to do so.

Suggestions?

Answer 1

vertices is a dynamic array of points, where each point is a float[3]. Try this example:

for (int i = 0; i < mesh.num_vertices; ++i) {
  printf("%d: x=%f y=%f z=%f\n", i, mesh.vertices[i][0], 
      mesh.vertices[i][1], mesh.vertices[i][2]);
}

If you look at the memory layout, it would be x0 y0 z0 x1 y1 z1 etc, each of those a float.