Slow point cloud creation with D435

[nbubis]

Required Info
Camera Model	D435
Operating System & Version	Ubuntu 22.04
Kernel Version (Linux Only)	5.15.148-tegra
Platform	Jetson Orin
SDK Version	v2.54.2
Language	python

Issue Description

Creation of a point cloud from the depth image seems to take quite a long time on the D435. Following the code in the examples:

for pipeline_idx, pipeline in enumerate(self.pipelines):
    frame = pipeline.wait_for_frames()
    frame_list.append(frame)

for pipeline_idx, frame in enumerate(frame_list):
    depth_frame = frame.get_depth_frame()
    t0 = time.time()
    pointcloud = rs.pointcloud().calculate(depth_frame).get_vertices()
    pointcloud = np.asanyarray(pointcloud).view(np.float32).reshape(-1, 3)
    print("Pointcloud calculation time: ", time.time() - t0)

On the L515, the time printed (to calculate the pointcloud) is less than 2 msec. However, using the D435, the time is roughly 50(!) msec for creating the pointcloud. I'm using only the depth camera (color is turned off), with a 640p resolution.

How can I speed up the pointcloud processing time?

[MartyG-RealSense]

Hi @nbubis Point cloud generation can be a processing-intensive task because the calculations are performed on the computer's processing CPU rather than the camera hardware. On Jetson boards, the processing can be accelerated by installing the librealsense SDK with support for CUDA enabled, so that the processing work is offloaded from the CPU onto the Jetson's graphics GPU.

There are two approaches to enabling CUDA support in librealsense.

1. Install the SDK from dedicated Jetson packages.

Or:

2. Compile the SDK from source code with CMake and include the flag -DBUILD_WITH_CUDA=TRUE in the CMake build instruction.

Instructions for both a package installation and source code build on Jetson can be found at the link below.

https://github.com/intelRealSense/librealsense/blob/master/doc/installation_jetson.md

[nbubis]

@MartyG-RealSense - What I find curious is that the times for the L515 are less than a 25th of the time for the D435. Arguably, the 515 is taking in only 320p, but that should account for a x4 speed up, not x25! I'm also comparing to a hand rolled python loop on other cameras on the same machine, there is no reason for the calculation to take longer than 10 msec even at higher resolutions.

Should I be using rs.rs2_deproject_pixel_to_point instead?

[MartyG-RealSense]

The performance of the pc.calculate and rs2_deproject_pixel_to_point methods of pointcloud generation should be very similar, except that pc.calculate is a little more accurate when an aligned depth-to-color pointcloud is being generated.

The L515 lidar depth camera model's hardware can process 23 million depth points per second, so there may be an advantage there over the D435 stereo depth camera.

You could try the rs.rs2_deproject_pixel_to_point Python pointcloud script at #11744 to see how its speed compares to your pc.calculate script.

[MartyG-RealSense]

Hi @nbubis Do you require further assistance with this case, please? Thanks!

[nbubis]

@MartyG-RealSense It would be good to confirm that the times I'm seeing are reasonable. Any benchmarks you could provide would be helpful to understand that this is expected behavior.

[MartyG-RealSense]

There are not any benchmarks for point cloud ms performance speed, which can be dependent on the program being used for pointcloud creation and the capabilities of the hardware that the program is being run on. However, #9194 is a case where a 400 Series camera user was experiencing pointcloud generation speed of 60 ms (compared to your 50 ms).

[nbubis]

@MartyG-RealSense - this seems to suggest the issue is in the code, since the user was getting sub 1 ms values when they removed the line:

_depth_units = sensor->get_option(RS2_OPTION_DEPTH_UNITS).query();

This line seems to have been moved way back in 2021, so without debugging the library object it's difficult to know what's casuing the issue 2.54.x.

[MartyG-RealSense]

I don't have advice to offer about the 2021 code changes, unfortunately. I do apologize.