How to render a mesh without texture in pytorch3d?

Question

I am trying to render a mesh with only vertices and faces given. I want to use pytorch3d for speed but I cannot seem to produce and image without texture. This is the code for the rendering without texture:

import torch
from pytorch3d.structures import Meshes
from pytorch3d.renderer import (
    FoVPerspectiveCameras,
    MeshRenderer,
    MeshRasterizer,
    RasterizationSettings,
    SoftSilhouetteShader
)


def render_mesh(vertices, faces):
    device = vertices.device
    rasterizer = MeshRasterizer(
        cameras=FoVPerspectiveCameras(device=device),
        raster_settings=RasterizationSettings(
            image_size=256,
            blur_radius=0.0,
            faces_per_pixel=1,
        )
    )
    renderer = MeshRenderer(
        rasterizer=rasterizer,
        shader=SoftSilhouetteShader()
    )
    mesh = Meshes(verts=[vertices], faces=[faces])
    image = renderer(mesh).squeeze()
    return image[..., :3]

The output is a blank (white) image. I also tried to add a dummy texture (https://github.com/facebookresearch/pytorch3d/issues/51) it throws "IndexError: The shape of the mask [1, 3] at index 1 does not match the shape of the indexed tensor [1, 9976, 3] at index 1". Any ideas? Thanks a lot!

EDIT: I am using this code to render with generic python libraries:

def create_scene(vertices, faces):
    tri_mesh = trimesh.Trimesh(vertices, faces)
    mesh = pyrender.Mesh.from_trimesh(tri_mesh)
    scene = pyrender.Scene()
    scene.add(mesh)
    camera = pyrender.PerspectiveCamera(yfov=np.pi/3, aspectRatio=1)
    camera_pose = np.eye(4)
    camera_pose[1, 3] = -0.02
    camera_pose[2, 3] = 0.3
    scene.add(camera, pose=camera_pose)
    light = pyrender.SpotLight(
        color=np.ones(3), 
        intensity=0.5,
        innerConeAngle=np.pi/16.0,
        outerConeAngle=np.pi/6.0
    )
    scene.add(light, pose=camera_pose)
    return scene


def render(vertices, faces):
    scene = create_scene(vertices, faces)
    renderer = pyrender.OffscreenRenderer(400, 400)
    color, _ = renderer.render(scene)
    renderer.delete()
    return color

The render function returns an image that looks like this:

Can you describe more precisely the desired output? Do you want the mesh to be rendered same as opened in MeshLab, with no texture and only shading? — ihdv, May 27 '23 at 06:15
@ihdv yes, I am not interested in color, only shape. Thanks for the interest! — Bojack, May 27 '23 at 09:35
In that case you may want use a different shader, e.g., HardPhongShader, The silhouette shader only gives silhouette. Similar to what you have done with pyrender, you also need to set a light so that there is shading. Also make sure the mesh is within the camera view (The mesh should probably be on +z axis but I can't remember exactly). — ihdv, May 27 '23 at 16:18

score 0 · Answer 1 · answered Jul 02 '23 at 12:23

You cannot render without a shader (except the silhouette shader) and cannot use a shader without a texture. What you actually want to do is rendering with a mono-coloured texture. This is also what you are doing in your code that is not using pytorch3d.

import torch
from pytorch3d.structures import Meshes
from pytorch3d.renderer import (
    FoVPerspectiveCameras,
    MeshRenderer,
    MeshRasterizer,
    RasterizationSettings,
    SoftSilhouetteShader
)
# Import Textures
from pytorch3d.renderer.mesh.textures import Textures


def render_mesh(vertices, faces):
    device = vertices.device
    rasterizer = MeshRasterizer(
        cameras=FoVPerspectiveCameras(device=device),
        raster_settings=RasterizationSettings(
            image_size=256,
            blur_radius=0.0,
            faces_per_pixel=1,
        )
    )
    renderer = MeshRenderer(
        rasterizer=rasterizer,
        shader=SoftSilhouetteShader()
    )
    mesh = Meshes(verts=[vertices], faces=[faces])
    # Add mono-coloured texture in the same shape as the vertices
    mesh.textures = torch.ones_like(vertices)[None]
    image = renderer(mesh).squeeze()
    return image[..., :3]

The two new lines are

from pytorch3d.renderer.mesh.textures import Textures
...
mesh.textures = torch.ones_like(vertices)[None]

Actually, I ended up using a HardGouraudShader with PointLights in order for it to work. Your suggestion results in a blank image, even when I adjust the translation and rotation. However, I'm still trying to get the SoftSilhouetteShader to work, as I imagine it will be faster. Do you have any suggestions? — Bojack, Jul 03 '23 at 13:21
Can you try swapping out `mesh.textures = torch.ones_like(vertices)[None]` with `mesh.textures = torch.zeroes_like(vertices)[None]`? It will turn the color from white to black. — Nopx, Jul 08 '23 at 05:28

How to render a mesh without texture in pytorch3d?

1 Answers1