Few ideas from an ex-zoneminder team member.

[baudneo]

I'll start this by saying I was a part of the zoneminder team and am the person that took on the enormous task of creating a new ml system for zoneminder. I am just now starting to seriously explore frigate and have some ideas.

My first idea is something that has been hashed over before; onvif motion/ml events. I won't hash it out here. Suffice it to say, if we could work this stuff in, it would be great.

My 2nd idea is to separate the ml and NVR components. Allowing the ml to be run either locally on the same host as frigate or remotely (on LAN or over Internet). Basically, frigate would either send batch or single images via http/ws/RPC/Unix socket to the ml server, the server processes and returns data in a reply and frigate can then process the objects. Can also add HA/load balancing if more than 1 remote ml server is configured

My 3rd idea is to allow multiple detectors per camera. Send a request with the images and a list of model names/uuids. Frigate parses the responses. This would allow things like "if yolov8x sees a person, only alert if the yolo-nas-l model also sees a person in approx same place/zone".

These are things I implemented in zm ml stack and so very desperately want to see in frigate. With multi modal cameras, you could run object detection, face recognition, alpr all on the same event. The fast API ml server can keep models loaded in memory to keep things fast, etc.

This is just a basic idea, can expand and get into details if there is interest in it.

Is this something that sound interesting? Please let me know your thoughts!

[NickM-27]

My 2nd idea is to separate the ml and NVR components. Allowing the ml to be run either locally on the same host as frigate or remotely (on LAN or over Internet). Basically, frigate would either send batch or single images via http/ws/RPC/Unix socket to the ml server, the server processes and returns data in a reply and frigate can then process the objects. Can also add HA/load balancing if more than 1 remote ml server is configured

I believe you and I have discussed this before, but this would be too slow. Frigate runs object detection in real time on every frame where motion or an active object is detected, not on collections of frames after the fact. This is part of how frigate actively tracks objects in real time, which is fundamentally different to how many other NVR object detection implementations work.

My 3rd idea is to allow multiple detectors per camera. Send a request with the images and a list of model names/uuids. Frigate parses the responses. This would allow things like "if yolov8x sees a person, only alert if the yolo-nas-l model also sees a person in approx same place/zone".

we have put a framework in place and have already implemented some post processors for face recognition and ALPR for Frigate 0.16.
This would include remote post processors that can run separate from frigate itself.

[baudneo]

Nice! Just a few follow up questions.

• 0.16 will allow multi model per camera? Or just adds fec rec/alpr?
• What backends will be used for face? Ageitgey face-recognition/dlib? Deep face framework?
• what backends for alpr? Onnx custom stuff? Http calls to plate recognizer/rekor?

Edit And sorry, any thoughts on remote ml component.

[NickM-27]

0.16 will allow multi model per camera? Or just adds fec rec/alpr?

It is a bit different to how you are thinking of it. Frigate has organized it into different tasks, not models per camera. So there are the main detectors which you described, then we have the semantic search embedding models in 0.15.

In 0.16 we have added a new framework for post processing objects, as a real time (local only) and post processing (can be remote). This includes:

• face recognition feature (built on top of opencv LBPHFaceRecognizer). I previously spent a lot of time working with face embedding models and the results just were not as good, and the lower resource usage requirement is preferable as well.
• ALPR built on paddleocr with onnxruntime
• bird classification using a tflite model

There are currently no remote capable remote post processors implemented, but we have quite a few we are planning

[hagak]

For your post-process ideas, could it be possible for a post-process to be able to inject metadata back to the object. Use-case is have a post process do OCR on a license plate and tag the recording with the license plate. Similar could be for known people using face recognition.

[NickM-27]

Use-case is have a post process do OCR on a license plate and tag the recording with the license plate. Similar could be for known people using face recognition.

Like I said these features (face rec and ALPR) are already implemented for 0.16. But to answer your question, yes, there are APIs available to set metadata on the object. In this case it would be https://docs.frigate.video/integrations/api/set-sub-label-events-event-id-sub-label-post