V3 RGBD Node #1198
V3 RGBD Node #1198
Conversation
Serafadam
commented
Dec 18, 2024
•
Serafadam
commented
- Host RGBD Node
- Basalt fixes
- Using stock TBB for Basalt
- Added optional Kompute support for performing calculations on GPU devices with Vulkan support
Serafadam
force-pushed
the
v3_rgbd_node
branch
from
4634ec4 to
26fe959
Compare
src/pipeline/node/host/RGBD.cpp
Outdated
| // Get the color and depth frames | ||
| auto group = inSync.get<MessageGroup>(); | ||
| if(group == nullptr) continue; | ||
| if(!initialized) { | ||
| initialize(group); | ||
| } | ||
| auto colorFrame = std::dynamic_pointer_cast<ImgFrame>(group->group.at(inColor.getName())); | ||
| if(colorFrame->getType() != ImgFrame::Type::RGB888i) { | ||
| throw std::runtime_error("RGBD node only supports RGB888i frames"); | ||
| } | ||
| auto depthFrame = std::dynamic_pointer_cast<ImgFrame>(group->group.at(inDepth.getName())); | ||
|
|
||
| // Create the point cloud | ||
| auto pc = std::make_shared<PointCloudData>(); | ||
| pc->setTimestamp(colorFrame->getTimestamp()); | ||
| pc->setTimestampDevice(colorFrame->getTimestampDevice()); | ||
| pc->setSequenceNum(colorFrame->getSequenceNum()); | ||
| pc->setInstanceNum(colorFrame->getInstanceNum()); | ||
| auto width = colorFrame->getWidth(); | ||
| auto height = colorFrame->getHeight(); | ||
| pc->setSize(width, height); | ||
|
|
||
| std::vector<Point3fRGB> points; | ||
| // Fill the point cloud | ||
| auto* depthData = depthFrame->getData().data(); | ||
| auto* colorData = colorFrame->getData().data(); | ||
| // Use GPU to compute point cloud | ||
| pimpl->computePointCloud(depthData, colorData, points); | ||
|
|
||
| pc->setPointsRGB(points); | ||
| pcl.send(pc); | ||
| auto rgbdData = std::make_shared<RGBDData>(); | ||
| rgbdData->depthFrame = *depthFrame; | ||
| rgbdData->rgbFrame = *colorFrame; | ||
| rgbd.send(rgbdData); | ||
| } | ||
| } |
There was a problem hiding this comment.
Maybe best to detect for each output RGBD and PCL if it is connected anywhere and only make the PCL / RGBD message if it will be used to save resources.
| ImgFrame rgbFrame; | ||
| ImgFrame depthFrame; |
There was a problem hiding this comment.
I think that it would be best to make this in a similar way as MessageGroup is done for performance reasons.
(so these two would be pointers and serialization would require a specialization)
examples/python/RGBD/rgbd.py
Outdated
There was a problem hiding this comment.
Playing around with it, at least for me, 640x400 provides a better overall experience (the PCLs are dense enough but everything runs at 30 FPS)
Same goes for other examples.
| /** | ||
| * Output RGBD frames. | ||
| */ | ||
| Output rgbd{*this, {"rgbd", DEFAULT_GROUP, {{DatatypeEnum::RGBDData, true}}}}; |
There was a problem hiding this comment.
The bindings are missing for this one
| py::arg("autoCreateCameras"), | ||
| py::arg("presetMode") = StereoDepth::PresetMode::HIGH_DENSITY) | ||
| py::arg("presetMode") = StereoDepth::PresetMode::HIGH_DENSITY, | ||
| py::arg("size") = std::pair<int, int>{1280, 720}) |
There was a problem hiding this comment.
Let's default to 1280x800 instead, it's the most common resolution for mono cameras.
examples/python/RGBD/rgbd_o3d.py
Outdated
There was a problem hiding this comment.
The pointcloud seems to be upside down for me at the beginning, is it upright on your side?
There was a problem hiding this comment.
It's in RDF frame convention, maybe that's why?
| dai::Pipeline pipeline; | ||
| auto rgbd = pipeline.create<dai::node::RGBD>()->build(true); | ||
|
|
||
| auto outQ = rgbd->pcl.createOutputQueue(); |
There was a problem hiding this comment.
Maybe we can add a test for rgbd as well
