[WIP] 🦀 Standalone Rust Inference

[ashwinvaidya17]

📝 Description

• Requires a lot of cleanup but here is the scaffolding.
• Produces simple heat map overlay

✨ Changes

Select what type of change your PR is:

• 🐞 Bug fix (non-breaking change which fixes an issue)
• 🔨 Refactor (non-breaking change which refactors the code base)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📚 Documentation update
• 🔒 Security update

✅ Checklist

Before you submit your pull request, please make sure you have completed the following steps:

• 📋 I have summarized my changes in the CHANGELOG and followed the guidelines for my type of change (skip for minor changes, documentation updates, and test enhancements).
• 📚 I have made the necessary updates to the documentation (if applicable).
• 🧪 I have written tests that support my changes and prove that my fix is effective or my feature works (if applicable).

For more information about code review checklists, see the Code Review Checklist.

[alexriedel1]

Is it fast?

[ashwinvaidya17]

Is it fast?

I haven't benchmarked this yet but I can say that this isn't the fastest. It uses the input image to set model input tensor info. But I feel we can make it more efficient by using Rust's native async feature to separate data fetching and model inference. Especially given that OpenVINO also supports async inference. This is more of a PoC for now. Not sure if we will go with Rust or C++ for standalone inference.

[alexriedel1]

Is it fast?

I haven't benchmarked this yet but I can say that this isn't the fastest. It uses the input image to set model input tensor info. But I feel we can make it more efficient by using Rust's native async feature to separate data fetching and model inference. Especially given that OpenVINO also supports async inference. This is more of a PoC for now. Not sure if we will go with Rust or C++ for standalone inference.

Alright! Thanks for the info :)