d435i pointcloud

[yash7321]

1)i am trying to do slam with only d435i in python not using ros , what will be my best approach here
and 2)is there a way to directly get pointcloud data from camera and use it to visualize live in open3d?

[MartyG-RealSense]

Hi @yash7321 The links below are a few examples of Python SLAM tools, though they are quite old and not tested with RealSense cameras.

https://github.com/DanielsKraus/SLAM-python
https://github.com/gisbi-kim/PyICP-SLAM
https://github.com/uoip/rgbd_ptam

isl-org/Open3D#473 may be a helpful reference for generating a RealSense pointcloud in Open3D.

[yash7321]

the rgb and depth image i get from real sense{get_depth_frame() and .get_color_frame()} do i need to preprocess them like
1)undistort and rectifying image using calibrated camera data(intrensic and extrensic )
2)using disparity map to compute depth map etc

and how can i get more accurate data for depth image ?

[MartyG-RealSense]

You do not need to perform rectification on the images as they are rectified by hardware inside the camera called the Vision Processor D4 before they are sent through the cable to the computer.

In regard to transforming disparity to depth, the Python reference at #11015 (comment) may be helpful.

Accuracy can be negatively affected by a range of different factors, including environment and lighting and the distance of the camera from the observed object / surface. In general though, using the Medium Density camera configuration preset will provide a good balance between accuracy and the amount of detail on the depth image.

You can use the Python instruction rs.option.visual_preset to set the Medium Density preset from the list of presets available. #10014 has an example of doing so. On the following line:

self.profile.get_device().sensors[0].set_option(rs.option.visual_preset, HIGH_DENSITY)

change HIGH_DENSITY to MEDIUM_DENSITY.

[yash7321]

Thank you for quick response, I am new to vision processing , it would be of great help if you could help with following

1. Direct Data Availability: Does the Intel RealSense D435i provide RGB images, depth images, and IMU data directly in a form that does not require further processing?
2. Synchronization of Data: Are the RGB images, depth data, and IMU data from the D435i synchronized?
3. Noise Reduction in Depth Maps: Is there a need for noise reduction on depth maps from the D435i? If so, what are the best methods for doing this?
4. Conversion of Depth Map to Point Cloud: What are the methods for converting depth maps to point clouds with greater accuracy?
5. SLAM with Point Cloud and IMU Data: Is it possible to use the point cloud and IMU data from the D435i for SLAM? What are the best approaches for this?

[MartyG-RealSense]

1. and 3. RealSense has a Low-Level Device API for accessing the sensors directly.

https://dev.intelrealsense.com/docs/api-architecture#low-level-device-api

More information about the Low-Level API can be found at IntelRealSense/realsense-ros#1409 (comment) as well as information relevant to question 3. about reducing depth noise.

2. Depth-RGB sync on the same camera should kick in when both streams have the same FPS value. If auto-exposure is enabled and an RGB option called Auto-Exposure Priority is disabled then the RealSense SDK will aid depth-RGB sync by enforcing a constant FPS for the two streams instead of permitting the FPS of one of the streams to vary, as confirmed at Wrong Color frame relative to Depth  #5675

If the instruction wait_for_frames() is used in a program script then the RealSense SDK should also try to find the closest timestamp match between the depth and RGB streams.

In regard to IMU, each IMU data packet is timestamped using the depth sensor hardware clock to allow temporal synchronization between gyro, accel and depth frames.

4. There are two main approaches to generating a pointcloud.

(a) Aligning depth to color and then using an instruction called rs2_deproject_pixel_to_point() to convert 2D pixels into 3D world points.

(b) Using the instructions map_to() and pc.calculate() to map depth and RGB together and generate a pointcloud.

map_to / pc.calculate should be a slightly more accurate method of generating a pointcloud than alignment and deprojection, though the difference is not significantly high.

5. Intel have a RealSense SLAM guide for producing a pointcloud with the IMU, though it is very old now.

https://github.com/IntelRealSense/realsense-ros/wiki/SLAM-with-D435i

[yash7321]

I have to do this whole process in python itself, will it be possible , if not which extra sensor to use ,to achieve my purpose of doing slam

[MartyG-RealSense]

Below are a few examples of Python SLAM projects, though none of their documentations specifically mention making use of an IMU.

https://github.com/gisbi-kim/PyICP-SLAM

https://github.com/DanielsKraus/SLAM-python

https://github.com/uoip/rgbd_ptam

[yash7321]

which post processing algorithms(and in what order) does real sense viewer use for showing depth map?

[MartyG-RealSense]

The recommended order for applying a list of post-processing filters is shown at the link below.

https://dev.intelrealsense.com/docs/post-processing-filters#using-filters-in-application-code

The RealSense Viewer tool has a range of post-processing filters enabled by default. The enabled by default ones have a blue icon beside them.

Depth data will still be displayable even if no post-processing filters are applied though.

[yash7321]

can i get depth map from realsence viewer directly after all this postprocessing to use in my code?

[MartyG-RealSense]

You could export a bag file from the Viewer, which is like a video recording of camera streams, and then import the bag file into your script. The script will then use the bag file data as though it is a real live camera.

You cannot save post-processing to a bag file though, so you would have to first load the bag file and then apply the post-processing filters to the bag's data in real-time. #1672 (comment) has a Python example of applying post-processing filters to depth data.

[yash7321]

when i get depth image from depth_frame = frames.get_depth_frame()
it has a shadow effect on left side, this shadow persists even after
decimation,spatial,temporal,hole_filling,threshold
i have attached the code , please go through it and where i am making mistake for this shadow effect to persist
code.txt

[MartyG-RealSense]

If the shadow that you describe is a vertical strip of black empty space on the left edge of the image then this is called an Invalid Depth Band. The phenomenon is described on pages 87-88 of the current edition of the data sheet document for the 400 Series cameras. You cannot remove this band but it is normal and harmless.

https://dev.intelrealsense.com/docs/intel-realsense-d400-series-product-family-datasheet

If the shadow is on the left side of an object or person, like in the above image, then this phenomenon is an occlusion. This is also normal and not something that needs to be corrected.

[yash7321]

thank you so much for clarifying this.

[yash7321]

align = rs.align(rs.stream.color)
frames = pipeline.wait_for_frames()
aligned_frames = align.process(frames)
depth_frame = aligned_frames.get_depth_frame()
color_frame = aligned_frames.get_color_frame()

in above code , the depth frame i get is wrt colour frame right? (that is from same point)
here i dont have to use inrensics rigth?

and if i want to create point cloud from that depth image , do i need to use any intrensics or extrensics, if so what values do i need to use?

[MartyG-RealSense]

The above code aligns the depth image to the color image. This means that the depth field of view size is resized to match the field of view size of the color stream, and the depth coordinates are mapped onto corresponding color coordinates.

It is not necessary to use intrinsics in this alignment code when a pointcloud is not being generated, as demonstrated by the RealSense SDK's align_depth2color.py Python alignment example program at the link below.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/align-depth2color.py

Intrinsics will have to be used when generating a pointcloud, like in the opencv_pointcloud_viewer.py example.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/opencv_pointcloud_viewer.py

#11031 (comment) provides a few different methods for producing a pointcloud with Python code using intrinsics.