Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rockchip: update dependencies and add script for model conversion #15699

Open
wants to merge 2 commits into
base: 0.16
Choose a base branch
from

Conversation

MarcA711
Copy link
Contributor

Proposed change

This PR updates the following rockchip dependencies:

  • FFmpeg 6.1.5 to 6.1.6 (includes a couple of bug fixes during transcoding)
  • rknn-toolkit-lite2 from v2.0.0 to v2.3.0
  • yolonas models were updated and converted using toolkit v2.3.0

Moreover, the image supports converting models from onnx to rknn format now. This should be useful for Frigate+ users who want to use their own models. Changes:

  • the script conv2rknn.py was added
  • python package rknn-toolkit2 was installed
  • 20 images from the COCO dataset are bundled with the image (rknn-toolkit2 needs these during quantization).

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code
  • Documentation Update

Additional information

Some notes about yolonas:
It seems the super-gradients project is not active anymore after Deci AI was bought by Nvidia. The current pypi release of super-gradients is broken because the download URL changed (see Deci-AI/super-gradients#2057). There are a couple of fixes:

  1. Download the weights manually from the new url (https://sg-hub-nv.s3.amazonaws.com/models/yolo_nas_s_coco.pth) and save them under ~/.cache/torch/hub/checkpoints/./.cache/torch/hub/checkpoints/
  2. Manually fix the package by following these instructions
  3. Install super-gradients from this fork that merged the fixes.

This is relevant for users who try to use this notebook from the frigate repo:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • The code has been formatted using Ruff (ruff format frigate)

Copy link

netlify bot commented Dec 27, 2024

Deploy Preview for frigate-docs ready!

Name Link
🔨 Latest commit c4e1191
🔍 Latest deploy log https://app.netlify.com/sites/frigate-docs/deploys/676ed1b82be41c00089d9217
😎 Deploy Preview https://deploy-preview-15699--frigate-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@NickM-27
Copy link
Collaborator

I think it would be best to target 0.16 with these changes, let me know what you think

@MarcA711
Copy link
Contributor Author

I agree

@NickM-27 NickM-27 changed the base branch from dev to 0.16 December 27, 2024 17:24
@NickM-27
Copy link
Collaborator

Are there any limitations on what model architectures can be converted to rknn format?

@MarcA711
Copy link
Contributor Author

On AArch64 only onnx to rknn is supported. This might change in the future.

On x86 other formats are available as well.

@NickM-27
Copy link
Collaborator

Sorry I mean are there any restrictions for example only yolonas, or also mobilenet and others?

@MarcA711
Copy link
Contributor Author

Sorry, I misunderstood.

In theory one can convert any model. Practically it's not a straightforward process. Not all onnx operations are supported.

For example, the default yolo-nas model includes post-processing and performes a batched non maximum suppression. The rknn-toolkit2 doesn't support this, so I had to modify the model and do the NMS programmatically in python.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants