How does Foxglove do calibration under the hood?

[henry-buron]

Specifically, I'm referring to when you associate a CameraInfo topic with a raw Image topic. The expected behavior is that, assuming the intrinsics and distortion coefficients are correct, Foxglove handles the undistortion/rectification of the raw image.

Does anyone know how this works under the hood? Is Foxglove using OpenCV undistort() or perhaps image_proc?

The reason I'm asking is because I was experiencing an issue where, in Foxglove, when I applied the calibration, there was still noticeable distortion in the "calibrated" image. However, when I used image_proc with the same CameraInfo and Image topic, the distortion was handled correctly.

Has anyone experienced a similar issue? The camera I was using was an Oak-D Pro W PoE.

[jtbandes]

Could you please share some data including your calibration parameters so we can look into this? Including steps/code to reproduce your results with image_proc would also help.

[henry-buron]

Hey @jtbandes,

Thanks for getting back to me. I have created a minimal example.

I have created a rosbag (ros2 humble) with an Image topic and CameraInfo topic. Please view the Image topic in Foxglove and calibrate with the CameraInfo topic. You should notice some distortion along the back wall throughout the video. Rosbag: https://drive.google.com/drive/folders/1EW-1oAyUyE5X47Xfwljhtz9BAK9TKO-4?usp=sharing

Then, run image_proc from the cli with: ros2 run image_proc rectify_node --ros-args -r image:=/perception/oak/raw/synced_left/image_raw -r camera_info:=/perception/oak/raw/synced_left/camera_info

This will publish an image_rect topic. View in Foxglove. The distortion should be removed.

Let me know if you have any questions or difficulty accessing the data. I'd also be interested in if you are able to replicate the problem on your end, or if I'm doing something wrong. Note: the camera is an Oak-D Pro W PoE. Perhaps the wide angle has something to do with it?

Thanks,
Henry

[jtbandes]

Thanks. To be clear, are you talking about distortions like these? Or is there something else you are seeing? (Overall, the lines in the image look straight to me, but I'm just eyeballing)

It's possible this is just an artifact of how we warp the image by applying the distortion correction to a relatively low-resolution mesh and not directly to each pixel.

I've created an internal ticket to track this and we'll look into it to see what improvements might be needed. https://linear.app/foxglove/issue/FG-10542

[henry-buron]

Yes, correct. Although in some frames the distortion definitely looks even worse. I've attached an image below (not from the rosbag) where the distortion is more noticeable.

The basic problem is that Foxglove showed distortion, indicating that the intrinsics/distortion coefficients were incorrect, while in reality they weren't.

[henry-buron]

One more example. Same calibration applied, but when viewed in Foxglove, a distortion artifact appears.