Question for using the Intel D405 as a tracking camera

[HanzDieter]

Hello,
I have a Intel D405 Stereo-Camera and I want to use this camera as a tracking camera with depth informationen.
I use this camera in a low textured enviroment at a very slow movement of the camera <200 mm/ minute. (Moving past a wall at a working distance of 200 mm)
With good lighting conditions I get very accurate depth-maps.
My questions are:

1. Do I have access of the integrated stereo-matching algorithm of the D405, which finds out corresponding points between the left and right image. I want to use this algorithm for finding correspoding points on two consecutive frames of the left camera of the D405 - so I get pixel shifts between this to consecutive frames / corresponding points, which I can use for getting the movement of the camera. (The intention behind is: The stereo-matching works pretty fine for generating a depth map of the enviroment, why should I not use this algorithm for determining the movement of the camera? - With algorithms like LucasKanade I do not get good results, because of the low textured enviroment.)
2. Is it possible to add an IMU to the D405, like the BNO085? Does the Realsense SDK provide a Sensorfusion and Calibration algorithm to merge the D405 with the IMU?
3. to reach I higher accuracy of the depth map I need a higher magnification of the details of my enviroment / the wall. Can I add a macro lens / tele converter / supplementary lens in front of the camera lenses to get a higher magnification? Then I need to recalibrate the camera and change the parameters of the D405. Are this steps possible with the Realsense SDK?

best regards,
Hanz

[MartyG-RealSense]

Hi @HanzDieter Thanks very much for your questions.

1. The stereo algorithms of RealSense cameras are not publicly available unfortunately as they are not open-source. Intel provide a detailed guide to implementing image-based tracking of motion in the guide at the link below.

https://dev.intelrealsense.com/docs/high-speed-capture-mode-of-intel-realsense-depth-camera-d435

The D405 camera model does not support the 300 FPS High Speed Capture Mode described in the paper though, as its fastest supported depth speed is 90 FPS.

2. If you need to integrate an IMU with the D405 for tracking purposes and want to be able to control it through the RealSense SDK, you could consider pairing the D405 with a RealSense T265 Tracking Camera. This is a camera whose features are dedicated to tracking and SLAM navigation instead of depth (as it does not have built-in depth sensing). It contains the BMI055 IMU.

Whilst the RealSense SDK supports T265 and it is still available in the RealSense Store at the time of writing this, it is now a retired camera model and so may not be available in the future.

https://store.intelrealsense.com/buy-intel-realsense-tracking-camera-t265.html

https://www.intelrealsense.com/how-to-getting-imu-data-from-d435i-and-t265/

https://www.intelrealsense.com/depth-and-tracking-combined-get-started/

https://dev.intelrealsense.com/docs/tracking-camera-t265-datasheet

---

If you would prefer to use your own choice of IMU instead though, then you should be able to make use of a D405 and data from an external IMU if you use ROS2 with the RealSense ROS wrapper ros2-development.

https://github.com/IntelRealSense/realsense-ros

If you would need to use ROS1 instead of ROS2, whilst the RealSense ROS1 wrapper does not support D405 it is possible to edit its code to add support, as described at IntelRealSense/realsense-ros#2582 (comment)

The RealSense SDK does have a Python-based IMU calibration tool but it is designed for use with built-in RealSense IMUs rather than an external one.

https://github.com/IntelRealSense/librealsense/tree/master/tools/rs-imu-calibration

Some RealSense users use a tool called Kalibr for their IMU calibration.

https://roboticsknowledgebase.com/wiki/sensing/camera-imu-calibration/

3. I once asked my Intel RealSense colleagues about whether adding a magnifying lens over the camera would work as a means of providing zoom and their answer was no. You could though move the camera closer to the observed surface though. By default the D405's minimum distance is 7 cm, but you can reduce this to 4 cm using the RealSense SDK's Disparity Shift option, setting it to a value of '100'. Doing so with a D405 is described at Sensor stability #10963 (comment)

This feature requires RealSense SDK version 2.51.1 or newer.

[HanzDieter]

Hello MartyG,
thank you very much for your answers :-)

1. so there is no function in the Realsense SDK, called e.g. stereo-matching(img1, img2), which uses the implemented stereo-algorithm?

What can I do with this function? I thought it is a simulation of the Realsense stereo-camera? (https://github.com/IntelRealSense/librealsense/tree/master/examples/software-device) Is it possilble with this emulation to do the job?

I will look at your suggestion, hoping I´ll get good results
2. I think that the T265 will not work in this low textured enviroment, because of their fisheye lenses, which cannot see the details, the D405 can? Or is only the IMU of the T265 used, when pairing them?
I will have a look at the ros-stuff you have provied - thank you :-)
3. unfortunately my working distance is fixed, so I can´t get any closer. One possibility would be using a magnifying lens and get only the image of the left and right camera. The stereo-matching is done by another algorithm, which runs on the PC - right?

Best regards,
Hanz

[MartyG-RealSense]

1. RealSense 400 Series cameras do have stereo matching but it is automated, and they do not have a 'confidence threshold' like the older RealSense SR300 / SR305 models do. On 400 Series cameras, pixels are either 100% matched or they are not matched. You can though simulate a condidence threshold using the Advanced Mode Second Peak Threshold option to allow or disallow depth coordinates based on the confidence in their accuracy.

The software-device feature allows you to define a 'virtual camera' and have the SDK use it to simulate a camera so that you do not need to have a real physical camera attached. It can also be used to import non-RealSense camera data from OpenCV such as webcam RGB into the SDK. software-device works best with C++ language though as it has problems in Python.

As the D405's camera hardware is designed specially for high accuracy, high quality results at close range, software-device would likely not provide a better close-range image than D405.

2. The fisheye sensors of the T265 are equipped with IR Cut filters like the D405 also is, so neither camera can see near-infrared frequencies such as the infrared dot pattern that other RealSense camera models can project. They can only see visible light frequencies.

If a surface has low texture or no texture, you could add texture to surfaces to aid depth analysis by using an external visible pattern projector that the D405 will be able to see and make use of. The section of Intel's guide about projectors linked to below has more information about this.

https://dev.intelrealsense.com/docs/projectors#6-the-consumer-front-projector

In regard to the IMU and pairing a T265 with D405, the T265 IMU would be used as D405 is not equipped with an IMU. This is not a problem though if the D405 and T265 were mounted together on a bracket, as both cameras will move in the same direction simultaneously when fixed on a mount. So the T265's orientation will also be the orientation of the D405 if they are both facing in the same direction on the mount.

3. Stereo matching (specifically, Semi-Global Matching) is done inside the camera hardware by a component called the Vision Processor D4. The functions performed by the Vision Processor D4 are listed in a PDF document at the link below.

https://www.stemmer-imaging.com/media/uploads/cameras/13/136603-Intel-RealSense-D4-Vision-Processor-Brochure.pdf

As mentioned above, my colleagues thought that a magnifying lens would not be helpful but you are completely free to experiment, of course.