How to create a basic gui with pointcloud updates using Open3D?

Question

I'm updating a point cloud from a depth camera stream and I'd like to create a basic UI for a bounding box to crop the point cloud.

I've made progress using open3d.visualization.VisualizerWithKeyCallback but I haven't figured out a way to get access to the window it creates. (VisualizerWithKeyCallback's create_window() simply returns a boolean).

I've also tried adapting the Open3D video example, however it's unclear how I would adapt the add_geometry() / update_geometry() approach I used with VisualizerWithKeyCallback as Open3DScene works differently.

For example, here's how I've attempted to add the pointcloud:

def _update_thread(self):
        # This is NOT the UI thread, need to call post_to_main_thread() to update
        # the scene or any part of the UI.
        while not self.is_done:
            time.sleep(0.100)

            if new_frame_is_valid:
                self.point_cloud_o3d.points = o3d.utility.Vector3dVector(points_np)

                
            # Update the images. This must be done on the UI thread.
            def update():
                if self.frame_count == 1:
                    self.widget3d.scene.add_geometry('original point cloud', self.point_cloud_o3d, self.lit)
                    print('added geometry')
                
            if not self.is_done:
                gui.Application.instance.post_to_main_thread(
                    self.window, update)

            self.frame_count += 1

This results in it appears to be blank window which turns black whenever I try to zoom/rotate.

Additionally I've thought about using the multiple_windows example however this simply crashes with my setup (Windows 11, Python 3.6.4, Open3D 0.15.1).

What's the simplest method of creating a basic UI using Open3D using a point cloud that needs constant updating (e.g. from a live depth camera) ?

George Profenza · Accepted Answer · 2022-09-15T16:46:10.673

I found a working answer in this Open3D github issue.

For reference, here's a modified version:

import numpy as np
import open3d as o3d

class Viewer3D(object):

    def __init__(self, title):
        app = o3d.visualization.gui.Application.instance
        app.initialize()

        self.main_vis = o3d.visualization.O3DVisualizer(title)
        app.add_window(self.main_vis)
        
        self.setup_depth_streaming()
        self.setup_point_clouds()
        self.setup_o3d_scene()

    def setup_depth_streaming(self):
        # TODO: setup your depth / point cloud streaming source here
        pass

    def setup_point_clouds(self):
        # setup an empty point cloud to be later populated with live data
        self.point_cloud_o3d = o3d.geometry.PointCloud()
        # the name is necessary to remove from the scene
        self.point_cloud_o3d_name = "point cloud"
        
    def update_point_clouds(self):
        # update your point cloud data here: convert depth to point cloud / filter / etc.
        pass

    def setup_o3d_scene(self):
        self.main_vis.add_geometry(self.point_cloud_o3d_name, self.point_cloud_o3d)
        self.main_vis.reset_camera_to_default()
        # center, eye, up
        self.main_vis.setup_camera(60,
                                    [4, 2, 5],
                                    [0, 0, -1.5],
                                    [0, 1, 0])

    def update_o3d_scene(self):
        self.main_vis.remove_geometry(self.point_cloud_o3d_name)
        self.main_vis.add_geometry(self.point_cloud_o3d_name, self.point_cloud_o3d)

    def run_one_tick(self):
        app = o3d.visualization.gui.Application.instance
        tick_return = app.run_one_tick()
        if tick_return:
            self.main_vis.post_redraw()
        return tick_return

viewer3d = Viewer3D("point cloud gui")

try:
    while True:
        # Step 1) Perturb the cloud with a random walk to simulate an actual read
        # (based on https://github.com/isl-org/Open3D/blob/master/examples/python/visualization/multiple_windows.py)
        viewer3d.update_point_clouds()
        # Step 2) Update the cloud and tick the GUI application
        viewer3d.update_o3d_scene()
        viewer3d.run_one_tick()
except Exception as e:
    print(e)

This is not perfect, but it's a start in terms of getting a point cloud to update and having a window reference (main_vis) to add controls to.

How to create a basic gui with pointcloud updates using Open3D?

1 Answers1