Replies: 20 comments 33 replies
-
|
I am currently using a N4100 soft router to run Frigate. When there is no one around, the page shows an inference speed of 20-30 milliseconds, but when there are people, it becomes 70-80 milliseconds. The actual feeling is that it takes two to three seconds for Home Assistant to get the information, which is too slow. I plan to purchase an Orange Pi 5, so I am particularly interested in the following two points: 1.Is an Orange Pi 5 with 4GB of memory enough to run Frigate? |
Beta Was this translation helpful? Give feedback.
-
|
Hi,
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your reply. I have tested before, triggering the frigate/<camera_name>/motion event is quite fast, and Home Assistant (HA) will receive the information very quickly. I haven’t done precise testing, but I feel it’s within a second. However, compared to the motion event, I feel that the object event will be much slower. So I previously thought that as long as the object detection is faster, the total time might still be within a second, then it can barely cooperate with other smart devices. But now that I think about it, as you said, there are other factors that affect it, making it difficult to respond reliably within a second. |
Beta Was this translation helpful? Give feedback.
-
|
Are these models really 320x320? I thought the YOLO NAS models were 640x640. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @blakeblackshear, One feature of the Rockchip platform is "Model Encryption" that could help to prevent license violation. However, I haven't used it so far and don't know how it works. But I thought it might be interesting. |
Beta Was this translation helpful? Give feedback.
-
|
Since the model is trained on 640x640, what does changing the input_size to 320x320 actually do? I'm considering adding YOLO-NAS as an available architecture for Frigate+ models. The models would then be compiled to ONNX format and that would enable broad support. Since they would be trained from scratch on a custom dataset, I can use them commercially. |
Beta Was this translation helpful? Give feedback.
-
|
Also, that export script in the README is a little different than the one on this page of the docs: https://docs.deci.ai/super-gradients/latest/documentation/source/models_export.html |
Beta Was this translation helpful? Give feedback.
-
|
I am not quite sure how pytorch adjusts the model to different input sizes. Here is the visualization of the first few layers of yolo-nas s 640x640 (left) and 320x320 (right): I didn't notice any difference comparing both models (however, I just tested detection on a few images).
I wasn't aware of this function. I will have a look how it works and if the exported models are better. |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure either. I would guess that specifying a size other than the size from training would just add a step that upscales the data to the original input, but I don't really know. |
Beta Was this translation helpful? Give feedback.
-
|
Frigate version: 0.14.0-b97e274 |
Beta Was this translation helpful? Give feedback.
-
|
Frigate version: 0.14.0-b97e274 |
Beta Was this translation helpful? Give feedback.
-
|
Frigate version: frigate:dev-d646338-rk |
Beta Was this translation helpful? Give feedback.
-
|
I have tried all deci-fp16-yolonas_* models. Trying to move from coral-based system. All of these models seem to weigh heavily on "cat". Same config (except for model and detectors). In 0.13.2 coral system, dogs were dogs, and people were people. Now dogs are always cats and people are sometimes cats. Rarely are there any actual cats. Are there any other models to try with the rockchip decoder? Should I just remove cat from object detection and see if that improves classification? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! Adding a labelmap fixed the issue with the cat/dog. I've only seen the cat/people issue once or twice. Appreciate the help! |
Beta Was this translation helpful? Give feedback.
-
|
Hey @NickM-27, frigate/frigate/detectors/plugins/rknn.py Line 42 in cf4517c Or will this not work (or even result in a race condition, where the labelmap is sometimes loaded before and sometimes after the change)? Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
See I think it should be okay |
Beta Was this translation helpful? Give feedback.
-
|
Im not sure I would do that programmatically. Probably best to put it in the docs and have users reference a different labelmap path in their config. I am looking at YOLONAS for Frigate+, so I wouldn't recommend baking an opinionated approach to the labelmap in the code. |
Beta Was this translation helpful? Give feedback.
-
|
I would only set this, if the user uses the rknn model that comes with frigate. If the user provides his own model, the default value |
Beta Was this translation helpful? Give feedback.
-
|
With the known issue of 8GB and 16GB rockchip boards having crashes, isn't this really a non-starter for frigate until this is fixed? |
Beta Was this translation helpful? Give feedback.
-
|
Well, my 16gb system runs multiple days without this issue. In my setup, it occurred just a few times. But my setup uses around 4GB and rarely more. I think if you run more stuff that requires more memory, you will likely run into this issue more often. Also, in order to reduce memory usage, you can set |
Beta Was this translation helpful? Give feedback.
-
|
At first, I figured this was the causing my issues with recordings, but then I found "Problem in 0.14.0 beta1... frigate stop recording every hour....needds to restart.. #11642". However, even if you aren't using more than 4GB, I would suspect that there is nothing telling the kernel to only use the first 4GB. It would seem that until they can fix the issue, it would be good if there is some kernel setting that would force it to only recognize the first 4GB. Do you know if this exists, or is my thinking flawed? |
Beta Was this translation helpful? Give feedback.
-
|
AFAIK, mpp requests an address within the first 4gb. The crash only happens, when there is no more space left. |
Beta Was this translation helpful? Give feedback.
-
|
With the beta v3 image, when restreaming 1024x768 mjpg stream from an esp32 camera module to m264 I'm getting about I'm 20% cpu usage on my rk3588. Feels like go2rtc ffmpeg is not using hardware acceleration yet. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, go2rtc doesn't support hardware acceleration for the FFmpeg fork used in frigate. See here for more information AlexxIT/go2rtc#979 However, you can use go2rtc's exec command like this: This transcodes h265 to h264 using hw accel and streams it to go2rtc. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for that! That should work with mjpeg as input stream as soon as their ffmpeg fork supports mjpeg decoding via mpp. Looks like that's on their to-do list. |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure if anyone knows but should the rknn detector just keep incrementally increase memory usage? It seems to go from 1% to about 8% usage after a day and then will flush back to 4% usage. Is this set somewhere in frigate? If I use a coral usb it stays around 2% usage so I wasn't sure if there was a reason it spikes so much higher vs the coral usb. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe there is a memory leak in the rknpu library. I haven't noticed this before but I will check if I can reproduce the issue. |
Beta Was this translation helpful? Give feedback.
-
|
Frigate version: 0.13.2-69d9a261 |
Beta Was this translation helpful? Give feedback.
-
|
Hey, thank you for posting your specs!
May I help you to get HW acceleration running? I guess it will work in the new frigate v14. |
Beta Was this translation helpful? Give feedback.
-
|
Sure, I would like to try to get it working. With |
Beta Was this translation helpful? Give feedback.
-
|
Could you post your Do you have something like this in it: Delete the empty |
Beta Was this translation helpful? Give feedback.
-
|
Here is my config after adding a detectors:
rknn:
type: rknn
model:
input_pixel_format: bgr
path: deci-fp16-yolonas_m
objects:
track:
- person
- car
- dog
- cat
- bicyle
- motorcycle
- airplane
- bus
mqtt:
enabled: false
record:
enabled: true
events:
retain:
mode: active_objects
ffmpeg:
hwaccel_args: preset-rk-h264 # or preset-rk-h265, depends on your cam
cameras:
porch:
ffmpeg:
inputs:
- path: rtsp://user:[email protected]/some-cam
roles:
- detect
- record
version: 0.14Frigate runs now, and $ sudo cat /sys/kernel/debug/rkrga/load
num of scheduler = 1
================= load ==================
scheduler[0]: rga2
load = 7%
-----------------------------------
process 20: pid = 249918, name: ffmpeg [many parameters...] |
Beta Was this translation helpful? Give feedback.
-
|
Yes you are right there is an issue. Normally Frigate should default to deci-fp16-yolonas_n if there is no path specifies, but it doesn't. I will fix this. Thank you for reporting. |
Beta Was this translation helpful? Give feedback.
-
|
Frigate version: 0.13.2-69d9a261 |
Beta Was this translation helpful? Give feedback.
-
|
Frigate version: 0.14.0-da913d8 |
Beta Was this translation helpful? Give feedback.
-
|
It seems that there is an incompatibility between the yolov8 models from Frigate v13 and newer kernels. This results in this error: I recommend upgrading to Frigate v14 since yolov8 is not supported anymore. If you encounter this problem in Frigate v14, please reply to this comment. This issue was initially reported here: nyanmisaka/ffmpeg-rockchip#95 |
Beta Was this translation helpful? Give feedback.
-
|
@MarcA711 Hi Marc, thanks for escalating this issue. I tested on the v14 rc2 - same result, reported here. It doesn't seem to be the problem related to the frigate version. Most likely the problem is between the RKNN toolkit and the model as suggested here. |
Beta Was this translation helpful? Give feedback.
-
Yes, this is also what I wrote above: However, upgrading frigate will also upgrade to newer models and eventually solve the problem. In your case, you have an old kernel with old rknn driver (v0.9.3). You should definitely upgrade to a newer version. I recommend this os: https://github.com/Joshua-Riek/ubuntu-rockchip |
Beta Was this translation helpful? Give feedback.
-
|
Hi @MarcA711 , @ALL, FYI: I tried the 14.0 release on the 6.1 image from here. Linux ubuntu 6.1.0-1021-rockchip #21-Ubuntu SMP Mon Jul 29 03:52:32 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux Good news - inference is working on one camera with good speed. I did not try with more cameras yet, this is planned. 
Frigate version: 0.14.0-da913d8 Still I was expecting better detector performance. My other setup is running on the pretty old Intel Core i7-6700K and inference there on ov detector is 10-20 ms. Given that the Rk3588 has the hardware acceelerator with 3 NPU cores and also hardware decoding of h.265 stream, I was expecting it to be at least on par with the old Core i7. Many thanks for your help and advice! |
Beta Was this translation helpful? Give feedback.
-
|
Frigate version: 0.14.0-da913d8
|
Beta Was this translation helpful? Give feedback.
-
|
just switched to new version Frigate version: 0.14.1-f4f3cfa great work |
Beta Was this translation helpful? Give feedback.
-
|
Thought I'd drop in my specifics after upgrading to latest version. Frigate version: 0.14.1-f4f3cfa Latest kernel lets me use NPU and Corals, or so it seems, not tested over time. |
Beta Was this translation helpful? Give feedback.
-
|
Just for the heck of it, I am currently running a test instance of 0.15.0b1. Heres the stats. Frigate version: 0.15.0-33825f6 When using default detection model, adding a PCI Coral with less error messages than on 0.14.1 |
Beta Was this translation helpful? Give feedback.
-
|
I will try to upgrade to 14. FriendlyELEC NanoPi M6 4gb marca711/frigate:latest-rk my config: mqtt: rtmp: |
Beta Was this translation helpful? Give feedback.
-
Could you share your Frigate config file on Rockchip for container. I did everything work ok but error on record |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for the late reply, I was busy during the last couple of days. |
Beta Was this translation helpful? Give feedback.
-
|
I'm investigating whether rk3588 would the right platform for my NVR use-case. Can you please share how many cameras (4k and 1080p) can rk3588 effectively handle with object detection turned on? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, The hardware has two things to take care of:
From the table above you can see that the rk3588's NPU needs around 30ms to perform one detection. This means that you can do around 35 detections per second. In frigates settings you can specify the frame rate for your detection stream. If you are interested to detect objects that are multiple seconds in the image (persons or slow moving animals) 1 FPS can be enough. If you want to catch quicker objects (for example fast driving cars) you might need to increase to 3 to 5 fps for reliable detection. If you want to detect a bird that passes your camera during his flight and is only visible for a split second you might need 15 to 30 fps for that. So here it depends how fast you want to detect objects. If you have 10 cameras that should only detect persons at 1 fps and 2 cameras that should be able to detect cars at 5 fps the npu needs to analyze up to 20 frames in one second. The rk3588 should be able to handle this. Now the VPU: Again, it depends on your setup. Usually, you have two streams per camera. One is a high resolution recording stream, the other one is a low resolution detection stream. To sum it up: So again, I haven't tested these numbers. But they are some theoretical limits of what the rk3588 can archive. If you have more questions about this or any other question, feel free to ask. You can also tell me about your setup (how many cams, what you want to detect at which frame rate, if you want to transcode your stream etc.) and I can tell you whether I think the rk3588 can handle this. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for the answer. Is any parallelisation / pipelining possible on rk3588's NPU? |
Beta Was this translation helpful? Give feedback.
-
|
I am not sure if I understand your question correctly. AFAIK frigate uses motion detection to select areas of your frame that might be interesting and adds it to a queue. The NPU processes the frames in the queue one by one. If the NPU is too slow to process all frames, frigate will drop old frames that haven't been processed. So it will not generate a hour long backlog that eats up your RAM and prevents object detection of newer frames if your NPU is too slow. But this is a software implementation in frigate and not really related to the rk3588's NPU. The NPU also has some sort of parallelism. If the NPU is used by frigate, I am still able to use the NPU from another process (for example from a python script that performs object detection of frames from my hard drive, even if I use a different ai model than frigate). So the NPU also has some way of handling this. But I don't know exactly how. |
Beta Was this translation helpful? Give feedback.
-
This is intended as an exchange for Rockchip users. We can talk about problems setting up hardware acceleration, optimizing performance, which distribution works best etc. Moreover, I want to collect some information like which OS + board works or not as well as inference times on different SoCs. I hope that as many users as possible help and submit data. Finally, I would also like to compile a list of FAQ, common pitfalls and known issues.
Latest news
The current beta version of Frigate includes lots of improvements in both hardware video processing and object detection for Rockchip devices. I would appreciate any help testing these changes. However, note that this is a beta version, so use with caution.
In this version, the option
core_maskof the rknn detector was replaced bynum_cores. You don't pass a complicated bit mask any more but just the number of cores you want to use (e.g.num_cores: 3for rk3588 SoCs).Moreover, Frigate Rockchip uses yolo-nas instead of yolov8. Yolo-nas is considered superior to yolov8. It is not suggested to stick to yolov8, since it is not supported anymore. Just changing the
model: path:to a yolov8 model will not work because Frigate does not implement post-processing for yolov8. However, keep in mind that yolo-nas is not suited for commercial usage, see this license.There was a kernel bug in version 6.1.0-1018-rockchip and 6.1.0-1019-rockchip of
ubuntu-rockchip's kernel. Upgrade to 6.1.0-1020-rockchip or above. To check your version, type:Docker image: ghcr.io/blakeblackshear/frigate:0.14.0-rc2-rk
Beta docs: https://deploy-preview-11419--frigate-docs.netlify.app/
Also see these notes for the beta release.
System compatibility
You can help complete this table by commenting these information:
Inference times
All times are in ms.
You can help complete this table by commenting these information:
FAQ
Is there a Frigate add-on for Home Assistant with support for Rockchip hardware?
No. There are (at least) 3 ways to install Home Assistant: their OS (HAOS), their supervised script (HA Supervised) and their docker image (HA Container). Only HAOS and HA Supervised support add-ons (see this comparison for more details).
HAOS uses the mainline Linux kernel that currently lacks the necessary drivers for Rockchip hardware acceleration (see this table to see the progress of mainlining the rk3588). HA Supervised currently lacks the option to unmask paths (
--security-opt systempaths=unconfined, see home-assistant/supervisor#4863). So there will be no add-on for Rockchip hardware until either HAOS or HA Supervised works.The only way to install Home Assistant and Frigate with Rockchip support is to use HA Container. For details see the next FAQ "How do I install Frigate and Home Assistant?". Also note that you don't need Home Assistant to use Frigate. Frigate already has a UI that received a major overhaul in v0.14. Using Home Assistant with Frigate makes only sense if you already use HA for other services and prefer an all-in-one solution or if you want to trigger other actions for certain events.
How do I install Frigate and Home Assistant?
This is currently only possible using HA Container (see previous FAQ "Is there a Frigate add-on for Home Assistant with support for Rockchip hardware?" for details). Note that you can't install add-ons in HA Container (see this comparison for more details). However, most add-ons are also available as docker images, you just need to configure them manually.
Please note:
The instructions below assume, that you are in an empty directory with read and write permission.
Now, create the files
docker-compose.ymlandmosquitto-data/config/mosquitto.confand paste the contents below. Thedocker-compose.ymlcreates a docker network and adds all containers to it. This way the containers can communicate with each other. Instead of an IP address you can use thecontainer_nameof each container.e.g.
nano docker-compose.ymle.g.
nano mosquitto-data/config/mosquitto.confNow start just the mosquitto container and add two users for Frigate and Home Assistant. Remember the password that you choose. Afterwards, create the
frigate-data/config/config.yml. I highlighted some lines that you might need to change.e.g.
nano frigate-data/config/config.ymlNow start Home Assistant and install HACS, afterwards start all containers:
Finally, you can open Home Assistant in your browser using
http://IP-of-your-server:8123. Go toSettings --> Integrations --> Add integration. There you can setup HACS, afterwards MQTT, afterwards Frigate (download it from HACS first). After installing HACS you should restart Home Assistant usingdocker compose restartin your terminal. During the setup of MQTT you are asked to enter the broker addressmosquittoas well as the usernamehomeassistantand password that you chose earlier. During the Frigate setup you need to enter the addresshttp://frigate:5000.You can stop your setup using
docker compose down, start usingdocker compose up -dand restart usingdocker compose restartif you are in the directory where you set everything up. Now, you should study the docs of each project and adopt everything to your needs.Known issues
Streams crash or recordings are missing
This is due to a hardware limitation that prevents the video processing hardware to access memory outside the first 4GB. This is potentially fixable, see rockchip-linux/mpp#560.
Beta Was this translation helpful? Give feedback.
All reactions