[Detector Support]: YoloNAS is abandoned - not a viable Yolov8/Yolov11 replacement

[aeozyalcin]

Describe the problem you are having

Super Gradients/Deci-AI got bought out by NVidia and they abandoned development on Yolo-NAS a few months ago. Their website with all documentation is unreachable (forwards to Nvidia.com), and their latest release doesn't even work out of the box, because they migrated all their servers, and the default URLs for pretrained Yolo-NAS models no longer work. The google colab notebook linked in the Frigate docs also fail, due to no longer working URLs.

I know that Yolo-NAS was introduced as an alternative to Ultralytics models. The licensing of Ultralytics models have been thoroughly discussed here and I know the developers of Frigate are hesitant to have anything to do with it.

So my question is, where do we go from here? I experimented extensively with YOLO-NAS (and still do, I have a way to access their latest servers), and it is an impressive model that could have sufficed as a Yolov8 replacement - had it not been for their acquisition and subsequently, abandonment of the GitHub repo. Yolox is hardly what I'd call SOTA model anymore, it falls well behind Yolov8/v11 and YoloNAS (RIP).

There is no technical reason why Yolov8/v11 couldn't work on Frigate - I know because I was the original developer who added yolox and yolov5 and v8 Openvino support to Frigate. My question for the developers: what is the progress/ETA on allowing custom detector "add-ons" so we, as the community, can re-support Ultralytics models on Frigate?

Version

v15

Frigate config file

NA

docker-compose file or Docker CLI command

NA

Relevant Frigate log output

NA

Install method

Docker Compose

Object Detector

OpenVino

Screenshots of the Frigate UI's System metrics pages

NA

Any other information that may be helpful

No response

[aeozyalcin]

Some references on YoloNAS being dead:
Deci-AI/super-gradients#2057
Deci-AI/super-gradients#2064
Deci-AI/super-gradients#2062

[NickM-27]

Frigate+ actively trains / offers yolonas models, so I'm not sure why you are counting it out as not viable.

Users have also recently been successful at downloading the pretrained models from the notebook in the docs. If this has stopped working again we can likely find a workaround.

CC @blakeblackshear

[aeozyalcin]

Try the notebook for yourself. Without manually changing the URLs in the supergradients source code, you can't get the pre-trained models.

I respect Frigate+, but not everyone is using it. I train my own models, and lack of documentation on Yolonas makes it unviable.

Try the link for yourself:
https://docs.deci.ai/super-gradients/latest/LICENSE.YOLONAS.html

[NickM-27]

Sure, I'm not suggesting everyone use Frigate+, but if Frigate+ yolonas model training is working I see no reason why it is not viable for other users as well. Given that I have no direct involvement, Blake may be able to shed more light on the documentation that he used and perhaps offer suggestions.

[aeozyalcin]

Thanks. Looking forward to @blakeblackshear 's input here. To be clear, my intention isn't to revive YoloNAS. It works enough with some hacks, but no longer has support from its development team (look at the last commit).

My main question & goal is, is there a plan to open up external add-ons/detector integrations? I could make local hacks to Frigate source-code to resupport yolov8/v11, but that's not viable and doesn't give back to the community.

[NickM-27]

As far as I know there is no current plan / that is not something we are currently working on. Realistically, I don't know if that is really needed anymore given that Frigate supports ONNX. You should be able to put whatever ONNX model you want and use it in Frigate. Perhaps the output tensor might be a different shape and need some tweaking

[aeozyalcin]

Yep that's exactly where I am getting stuck. The output tensor is a different shape, and that's something that needs to be taken care of in post-process, which is what the code I originally checked in did. I am not aware of/have the expertise to change the model itself.

[hawkeye217]

For reference: #14457

[aeozyalcin]

Yep. I am not saying it's impossible. That is in fact what I am also doing on my own workflow right now. It requires users to change the URL in the source code itself.

[blakeblackshear]

I don't have this issue with YOLO-NAS since I don't use the pretrained weights in Frigate+. The current version still works, but there is no evidence that it will be maintained going forward unfortunately. It's a bit of a pain because most of these are research projects and fall out of maintenance.

I am hoping this MIT licensed YOLOv9 rewrite gets some traction: https://github.com/WongKinYiu/YOLO. It's right up there with the Ultralytics models in the Hailo model zoo: https://hailo.ai/products/hailo-software/model-explorer/

There may be a way to add more generic support for yolo models in the ONNX or other detectors. Yolov8 might happen to work as a side effect, but we can't distribute a model. At the moment, I believe the yolov8 model from the Hailo model zoo does actually work with Frigate already.

[aeozyalcin]

Thanks Blake. I am not saying I am doing it, but theoretically if I were to allegedly do this, is there an elegant way of getting around the config validator which expects only ssd, yolox and yolonas types?

I already, allegedly, replaced the reference of ModelTypeEnum from frigate.detectors.detector_config with a locally defined one in my custom detector - allegedly. And the single bind in the docker compose is working pretty elegantly, allegedly, and the only issue is the config validator, when Frigate is starting, because the model type is not ssd, yolox or yolonas - allegedly.

[blakeblackshear]

Probably simplest to just use the existing options. It sounds to me like you are just doing a custom detector for a custom yolonas model that needs a special implementation. These enum values are just strings, so how you handle the model type in your detector is up to you.

[NonaSuomy]

I don't have this issue with YOLO-NAS since I don't use the pretrained weights in Frigate+. The current version still works, but there is no evidence that it will be maintained going forward unfortunately. It's a bit of a pain because most of these are research projects and fall out of maintenance.

I am hoping this MIT licensed YOLOv9 rewrite gets some traction: https://github.com/WongKinYiu/YOLO. It's right up there with the Ultralytics models in the Hailo model zoo: https://hailo.ai/products/hailo-software/model-explorer/

There may be a way to add more generic support for yolo models in the ONNX or other detectors. Yolov8 might happen to work as a side effect, but we can't distribute a model. At the moment, I believe the yolov8 model from the Hailo model zoo does actually work with Frigate already.

(I understand this is not support but just wanted to test the thought above out)

As I was getting inference times of 10ms with v8 compared to nas @ 90+ms

Log of it working with yolo-nas

2025-01-19 07:57:29.028798059  [2025-01-19 07:57:29] frigate.detectors.plugins.rocm WARNING : AMD/ROCm: detected gfx900 but HSA_OVERRIDE_GFX_VERSION already present (9.0.0), not overriding!
2025-01-19 07:57:29.075195084  [2025-01-19 07:57:29] frigate.app                    INFO    : Camera processor started for HQFrontYardCam: 412
2025-01-19 07:57:29.104765228  [2025-01-19 07:57:29] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: loaded migraphx module
2025-01-19 07:57:29.105212523  [2025-01-19 07:57:29] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: switching HIP to blocking mode to conserve CPU
2025-01-19 07:57:29.105443524  [2025-01-19 07:57:29] frigate.app                    INFO    : Capture process started for HQFrontYardCam: 419
2025-01-19 07:57:29.253064411  [2025-01-19 07:57:29] frigate.api.fastapi_app        INFO    : Starting FastAPI app
2025-01-19 07:57:29.568290086  [2025-01-19 07:57:29] frigate.api.fastapi_app        INFO    : FastAPI started
2025-01-19 07:57:30.353144149  [2025-01-19 07:57:29] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: loading parsed model from /config/yolo_nas_s.mxr
2025-01-19 07:57:30.353436679  [2025-01-19 07:57:30] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: model loaded
2025-01-19 07:57:35.489535408  [INFO] Starting go2rtc healthcheck service...

When I tried the other models it stops here because that is not a selectable model in the code. Is there a quick fix to force it to accept random model codes or you would have to recompile from source and add a lot of code to handle these other models?

2025-01-19 06:27:33.633170477  [2025-01-19 06:27:33] frigate.util.config            INFO    : frigate config does not need migration...
2025-01-19 06:27:33.654838932  *************************************************************
2025-01-19 06:27:33.654841601  *************************************************************
2025-01-19 06:27:33.654843433  ***    Your config file is not valid!                     ***
2025-01-19 06:27:33.654846325  ***    Please check the docs at                           ***
2025-01-19 06:27:33.654847494  ***    https://docs.frigate.video/configuration/          ***
2025-01-19 06:27:33.654849056  *************************************************************
2025-01-19 06:27:33.654850231  *************************************************************
2025-01-19 06:27:33.654872794  ***    Config Validation Errors                           ***
2025-01-19 06:27:33.654874064  *************************************************************
2025-01-19 06:27:33.654875028  model.model_type: Input should be 'ssd', 'yolox' or 'yolonas'
2025-01-19 06:27:33.654894329  *************************************************************
2025-01-19 06:27:33.654895423  ***    End Config Validation Errors                       ***
2025-01-19 06:27:33.654912781  *************************************************************
2025-01-19 06:27:33.993244266  [INFO] Service Frigate exited with code 1 (by signal 0)

I also tried to leave the model yolonas and it seem to load but then get this error:

2025-01-19 06:14:09.083122450  [INFO] Starting go2rtc healthcheck service...
2025-01-19 06:14:22.863993031  [2025-01-19 06:14:22] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2025-01-19 06:14:22.885175842  [2025-01-19 06:14:22] detector.rocm                  INFO    : Starting detection process: 502
2025-01-19 06:14:22.945640806  [2025-01-19 06:14:22] frigate.detectors.plugins.rocm WARNING : AMD/ROCm: detected gfx900 but HSA_OVERRIDE_GFX_VERSION already present (9.0.0), not overriding!
2025-01-19 06:14:23.016005677  [2025-01-19 06:14:22] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: loaded migraphx module
2025-01-19 06:14:23.016029213  [2025-01-19 06:14:22] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: switching HIP to blocking mode to conserve CPU
2025-01-19 06:14:24.112096723  [2025-01-19 06:14:23] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: loading parsed model from /config/yolov8s.mxr
2025-01-19 06:14:24.112363685  [2025-01-19 06:14:24] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: model loaded
2025-01-19 06:14:24.245782317  Process detector:rocm:
2025-01-19 06:14:24.245786369  Traceback (most recent call last):
2025-01-19 06:14:24.245788649    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2025-01-19 06:14:24.245790022      self.run()
2025-01-19 06:14:24.245791956    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
2025-01-19 06:14:24.245793387      return run(*args, **kwargs)
2025-01-19 06:14:24.245795241    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2025-01-19 06:14:24.245796699      self._target(*self._args, **self._kwargs)
2025-01-19 06:14:24.245798288    File "/opt/frigate/frigate/object_detection.py", line 145, in run_detector
2025-01-19 06:14:24.245799786      detections = object_detector.detect_raw(input_frame)
2025-01-19 06:14:24.245801360    File "/opt/frigate/frigate/object_detection.py", line 95, in detect_raw
2025-01-19 06:14:24.245802883      return self.detect_api.detect_raw(tensor_input=tensor_input)
2025-01-19 06:14:24.245804759    File "/opt/frigate/frigate/detectors/plugins/rocm.py", line 138, in detect_raw
2025-01-19 06:14:24.245806781      detector_result = self.model.run({model_input_name: tensor_input})[0]
2025-01-19 06:14:24.245812016  RuntimeError: /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/AMDMIGraphX/src/program.cpp:461: operator(): Incorrect shape {uint8_type, {1, 3, 640, 640}, {1228800, 409600, 640, 1}} for parameter: images should be: float_type, {1, 3, 640, 640}, {1228800, 409600, 640, 1}
2025-01-19 06:14:32.876092580  [2025-01-19 06:14:32] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting Frigate...

Tested the model made by harakas
input_pixel_format: bgr
model_type: yolox
width: 320 # match onnx settings
height: 320 # match onnx settings
path: /config/model_cache/yolov8/yolov8s_320x320.onnx
labelmap_path: /config/model_cache/yolov8/labels.txt

2025-01-19 07:49:42.011156322  RuntimeError: /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/AMDMIGraphX/src/program.cpp:461: operator(): Incorrect shape {uint8_type, {1, 3, 320, 320}, {307200, 102400, 320, 1}} for parameter: images should be: float_type, {1, 3, 320, 320}, {307200, 102400, 320, 1}

I thought that this maybe meant that the model was trained at 640 x 640 / 320 x 320 and you have to set those settings but alas it didn't fix it.

Are these the basic key ONNX models that should work?

Task: Instance Segmentation COCO

Network name	Task	Input Resolution	Operations	Parameters	License	Full Precision Accuracy	Post Quantization Accuracy	FPS (Batch Size=1)	FPS (Batch Size=8)	Retraining Docker
yolov8s	Object Detection COCO	640x640x3	28.6G	11.2M	AGPL-3.0	44.58	43.94	398.37	398.32	Yes
yolox_tiny	Object Detection COCO	416x416x3	6.44G	5.05M	Apache-2.0	32.64	31.37	216.76	563.65	Yes
yolox_s_leaky	Object Detection COCO	640x640x3	26.74G	8.96M	Apache-2.0	38.13	37.29	376.18	376.15	Yes
damoyolo_tinynasL20_T	Object Detection COCO	640x640x3	18.02G	11.35M	Apache-2.0	42.80	42.33	132.22	322.99	Yes
yolov9c	Object Detection COCO	640x640x3	102.1G	25.3M	GPL-3.0	52.60	51.10	36.78	65.56	No
yolov11n	Object Detection COCO	640x640x3	6.55G	2.6M	AGPL-3.0	39.00	37.49	156.90	398.94	No

I changed those config settings and still got the same error. What am I missing to load the ONNX models from the Hailo model zoo: https://hailo.ai/products/hailo-software/model-explorer/ ?

Model configs

model:
  #model_type: yolonas
  #width: 320 # match notebook settings
  #height: 320 # match notebook settings
  #path: /config/yolo_nas_s.onnx
  #labelmap_path: /labelmap/coco-80.txt

  input_pixel_format: bgr  
  model_type: yolov8
  width: 640 # Match onnx trained settings
  height: 640 # Match onnx trained settings
  path: /config/yolov8s.onnx  
  labelmap_path: /config/model_cache/yolov8/labels.txt

If you remove more things from the labelmap would inference times be any better? There is no way a giraffe is going to walk by my house. A lot of those 80 coco items seem a little out there in most generic cases.

Docker Compose

# Frigate Container for Docker
# Go here for updated repo numbers: https://github.com/blakeblackshear/frigate/pkgs/container/frigate
services:
  frigate:
    container_name: frigate
    privileged: true
    #restart: "no"
    restart: "unless-stopped"
    image: ghcr.io/blakeblackshear/frigate:3947e79-rocm
    #image: ghcr.io/blakeblackshear/frigate:0.13.0-beta5
    #image: ghcr.io/blakeblackshear/frigate:stable
    #image: ghcr.io/harakas/frigate:latest-rocm-gfx900
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
      - "com.centurylinklabs.watchtower.monitor-only=true"
    shm_size: "64mb"
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
      - /dev/dri
      - /dev/kfd
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/docker-storage/frigate/config/:/config
      #- /var/lib/libvirt/images/usb001/docker-storage/frigate/config/go2rtc:/config/go2rtc
      - /opt/docker-storage/frigate:/media/frigate
      #- /var/lib/libvirt/images/usb001/docker-storage/frigate:/config/db
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    hostname: "frigate"
    networks:
      dhcp:
        mac_address: "de:ad:be:ef:ca:fe"
    ports:
      - "5000:5000" # WebUI
      - "1935:1935" # RTMP feeds
      - "8554:8554" # Birdseye In v 0.12
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "password"
      HSA_OVERRIDE_GFX_VERSION: "9.0.0"
      #DOWNLOAD_YOLOV8: "1"
      #HCC_SERIALIZE_KERNEL=3
      #HCC_SERIALIZE_COPY=3
      #HSA_ENABLE_SDMA=0
      #HSA_ENABLE_INTERRUPT=0
      #HSA_SVM_GUARD_PAGES=0
      #HSA_DISABLE_CACHE=1

networks:
  dhcp:
    name: dbrv100
    external: true

config.yaml

  host: 10.0.1.42
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: frigate
  password: password

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p
#  input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000

go2rtc:
  streams:
    HQFrontYardCam:
      - rtsp://frigate:[email protected]:554/Streaming/Channels/101/
    HQFrontYardCam_sub:
      - rtsp://frigate:[email protected]:554/Streaming/Channels/102/

model:
  #model_type: yolonas
  #width: 320 # match notebook settings
  #height: 320 # match notebook settings
  #path: /config/yolo_nas_s.onnx
  #labelmap_path: /labelmap/coco-80.txt

  input_pixel_format: bgr  
  model_type: yolox
  width: 320 # match onnx settings
  height: 320 # match onnx settings
  path: /config/model_cache/yolov8/yolov8s_320x320.onnx
  labelmap_path: /config/model_cache/yolov8/labels.txt

detectors:
  #cpu1:
  #  type: cpu
  #  num_threads: 2
  #cpu2:
  #  type: cpu
  #  num_threads: 2
  rocm:
    type: rocm

#Global Object Settings
objects:
  track:
    - person
    - dog
    - cat
    - mouse
    - bird
    - car
    - bicycle
    - motorcycle
    - bus
  #filters:
   # person:
     # min_area: 5000
     # max_area: 100000

cameras:
  HQFrontYardCam:
    ffmpeg:
      inputs:
        # High Resolution Stream
        - path: rtsp://127.0.0.1:8554/HQFrontYardCam
          input_args: preset-rtsp-restream
          roles:
            - record
        # Low Resolution Stream
        - path: rtsp://127.0.0.1:8554/HQFrontYardCam_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 640
      height: 480
      fps: 5
    snapshots:
      enabled: true
      bounding_box: true
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      alerts:
        retain:
          days: 10
      detections:
        retain:
          days: 10
    motion:
      improve_contrast: true
      mask:
        - 214,15,214,31,25,31,25,15
    zones:
      zone_0_front_door:
        coordinates: 463,239,360,238,285,206,292,150,369,62,394,33,476,33
      zone_0_Driveway:
        coordinates: 385,228,79,147,204,77,396,64

birdseye:
  enabled: true
  restream: false
  width: 1280
  height: 720
  quality: 1
  mode: objects

live:
  height: 640
  quality: 1

version: 0.15-1

[duylong]

2025-01-19 06:14:24.245812016 RuntimeError: /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/AMDMIGraphX/src/program.cpp:461: operator(): Incorrect shape {uint8_type, {1, 3, 640, 640}, {1228800, 409600, 640, 1}} for parameter: images should be: float_type, {1, 3, 640, 640}, {1228800, 409600, 640, 1}
2025-01-19 06:14:32.876092580 [2025-01-19 06:14:32] frigate.watchdog INFO : Detection appears to have stopped. Exiting Frigate...

Same problem with Frigate 0.15. Do you have found a solution ? I saw an answer here (Deci-AI/super-gradients#1969 (comment)) that says that "having uint8 may indicate that we don't have any image normalization".

[duylong]

Ok, it works by strictly following the procedure https://colab.research.google.com/github/blakeblackshear/frigate/blob/dev/notebooks/YOLO_NAS_Pretrained_Export.ipynb

I finally backed down to edgetpu, because rocm is slow on my device (71.06 ms)

[NonaSuomy]

rocmunsupportedmod.zip
Note: This breaks the other models...
I got it running with this modified code. Maybe it will help someone get other models running. (I'm not sure if I broke it as I get detected as a dog with like 700+%) but should be a good starting point maybe as it compiles. 😆
Models come from here https://github.com/harakas/frigate/blob/dev/docs/docs/configuration/object_detectors.md#amdrocm-gpu-detector

To test the code, I connected to the container console and installed vim. Then dumped the code above. After restarting it worked for a bit then got this error...

2025-01-20 08:07:03.273682242  [2025-01-20 08:07:03] frigate.record.maintainer      WARNING : Too many unprocessed recording segments in cache for HQFrontYardCam. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...

2025-01-20 08:07:13.277791323  [2025-01-20 08:07:13] frigate.record.maintainer      WARNING : Too many unprocessed recording segments in cache for HQFrontYardCam. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...

2025-01-20 08:07:23.288736605  [2025-01-20 08:07:23] frigate.record.maintainer      WARNING : Too many unprocessed recording segments in cache for HQFrontYardCam. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...

model:
  #model_type: yolonas
  #width: 320 # match notebook settings
  #height: 320 # match notebook settings
  #path: /config/yolo_nas_s.onnx
  #labelmap_path: /labelmap/coco-80.txt

  #iput_pixel_format: bgr

  input_pixel_format: rgb
  model_type: yolov8
  width: 320 # Match onnx trained settings
  height: 320 # Match onnx trained settings
  path: /config/model_cache/yolov8/yolov8s_320x320.onnx
  labelmap_path: /config/model_cache/yolov8/labels.txt

detectors:
  #cpu1:
  #  type: cpu
  #  num_threads: 2
  #cpu2:
  #  type: cpu
  #  num_threads: 2
  rocm:
    type: rocm    

The inference time was back down to 10-20ms with v8 compared to 90ms with yolo-nas on an ancient AMD Vega64. That or I broke the reporting.