In short, Open3D expects your 3-channel color image to be of uint8 type.
Otherwise, it would return an empty point cloud, resulting in the blank window you see.
Update 2020-3-27, late night in my time zone:)
Now that you have provided your code, let's dive in!
From your function names, I guess you are using Open3D 0.7.0 or something like that. The code I provided is in 0.9.0. Some function names changed and new functionality added in.
When I run your code in 0.9.0 (after some minor modifications of course), there's a RuntimeError:
RuntimeError: [Open3D ERROR] [CreatePointCloudFromRGBDImage] Unsupported image format.
And we can see from the Open3D 0.9.0 source that your color image must be of 3 channels and take only 1 byte each (uint8) or be of 1 channel and take 4 bytes (float, that means intensity image):
std::shared_ptr<PointCloud> PointCloud::CreateFromRGBDImage(
const RGBDImage &image,
const camera::PinholeCameraIntrinsic &intrinsic,
const Eigen::Matrix4d &extrinsic /* = Eigen::Matrix4d::Identity()*/,
bool project_valid_depth_only) {
if (image.depth_.num_of_channels_ == 1 &&
image.depth_.bytes_per_channel_ == 4) {
if (image.color_.bytes_per_channel_ == 1 &&
image.color_.num_of_channels_ == 3) {
return CreatePointCloudFromRGBDImageT<uint8_t, 3>(
image, intrinsic, extrinsic, project_valid_depth_only);
} else if (image.color_.bytes_per_channel_ == 4 &&
image.color_.num_of_channels_ == 1) {
return CreatePointCloudFromRGBDImageT<float, 1>(
image, intrinsic, extrinsic, project_valid_depth_only);
}
}
utility::LogError(
"[CreatePointCloudFromRGBDImage] Unsupported image format.");
return std::make_shared<PointCloud>();
}
Otherwise, there'll be errors like I encountered.
However, in the version of 0.7.0, the source code is:
std::shared_ptr<PointCloud> CreatePointCloudFromRGBDImage(
const RGBDImage &image,
const camera::PinholeCameraIntrinsic &intrinsic,
const Eigen::Matrix4d &extrinsic /* = Eigen::Matrix4d::Identity()*/) {
if (image.depth_.num_of_channels_ == 1 &&
image.depth_.bytes_per_channel_ == 4) {
if (image.color_.bytes_per_channel_ == 1 &&
image.color_.num_of_channels_ == 3) {
return CreatePointCloudFromRGBDImageT<uint8_t, 3>(image, intrinsic,
extrinsic);
} else if (image.color_.bytes_per_channel_ == 4 &&
image.color_.num_of_channels_ == 1) {
return CreatePointCloudFromRGBDImageT<float, 1>(image, intrinsic,
extrinsic);
}
}
utility::PrintDebug(
"[CreatePointCloudFromRGBDImage] Unsupported image format.\n");
return std::make_shared<PointCloud>();
}
That means Open3D still does not support it, but it would only warn you. And only in debug mode!
After that, it will return an empty point cloud. (Actually both versions do this.) That explains the blank window.
Now you should know, you can make convert_rgb_to_intensity=True and succeed. Though you still should normalize your color image first.
Or you can convert the color image to be in range [0, 255] and of type uint8.
Both will work.
Now I hope all is clear. Hooray!
P.S. Actually I usually found Open3D source code to be easy to read. And if you know C++, you could read it whenever something like this happens.
Update 2020-3-27:
I cannot reproduce your result and I don't know why it happened (you should provide a minimal reproducible example).
Using the image you provided in the comment, I wrote the following code and it works well. If it still doesn't work on your computer, maybe your Open3D is broken.
(I'm not familiar with .exr images, hence the data extraction might be ugly :)
import Imath
import array
import OpenEXR
import numpy as np
import open3d as o3d
# extract data from exr files
f = OpenEXR.InputFile('frame.exr')
FLOAT = Imath.PixelType(Imath.PixelType.FLOAT)
cs = list(f.header()['channels'].keys()) # channels
data = [np.array(array.array('f', f.channel(c, FLOAT))) for c in cs]
data = [d.reshape(720, 1280) for d in data]
rgb = np.concatenate([data[i][:, :, np.newaxis] for i in [3, 2, 1]], axis=-1)
# rgb /= np.max(rgb) # this will result in a much darker image
np.clip(rgb, 0, 1.0) # to better visualize as HDR is not supported?
# get rgbd image
img = o3d.geometry.Image((rgb * 255).astype(np.uint8))
depth = o3d.geometry.Image((data[-1] * 1000).astype(np.uint16))
rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(img, depth, convert_rgb_to_intensity=False)
# some guessed intrinsics
intr = o3d.open3d.camera.PinholeCameraIntrinsic(1280, 720, fx=570, fy=570, cx=640, cy=360)
# get point cloud and visualize
pcd = o3d.geometry.PointCloud.create_from_rgbd_image(rgbd, intr)
o3d.visualization.draw_geometries([pcd])
And the result is:
Original answer:
You have misunderstood the meaning of depth_scale.
Use this line:
depth = o3d.geometry.Image(np.array(img_array[:, :, 3]*1000).astype('uint16'))
The Open3D documentation said the depth values will first be scaled and then truncated. Actually it means the pixel values in your depth image will first divide this number rather than multiply, as you can see in the Open3D source:
std::shared_ptr<Image> Image::ConvertDepthToFloatImage(
double depth_scale /* = 1000.0*/, double depth_trunc /* = 3.0*/) const {
// don't need warning message about image type
// as we call CreateFloatImage
auto output = CreateFloatImage();
for (int y = 0; y < output->height_; y++) {
for (int x = 0; x < output->width_; x++) {
float *p = output->PointerAt<float>(x, y);
*p /= (float)depth_scale;
if (*p >= depth_trunc) *p = 0.0f;
}
}
return output;
}
Actually we usually take it for granted that values in depth images should be integers (I guess that's why Open3D did not point that out clearly in their documentation), since floating-point images are less common.
You cannot store 1.34 meters in .png images, since they will lose precision. As a result, we store 1340 in depth images and later processes will first convert it back to 1.34.
As for camera intrinsics for your depth image, I guess there'll be configuration parameters in Blender when you create it. I'm not familiar with Blender, so I'll not talk about it. However, if you do not understand general camera intrinsics, you might want to take a look at this.
