[Detector Support]: Openvino and Intel integrated NPU support #13248

philippedegeer · 2024-08-21T15:20:29Z

philippedegeer
Aug 21, 2024

Describe the problem you are having

Hello,

I have been using Frigate 14 with YoloNAS-M (256x256) with great satisfaction. For my use case, I have no FP and an inference speed below 10ms with an Intel Core Ultra 7 155H with the iArc GPU.

I have been exploring the possibility to utilize the NPU, to offload the GPU from this task

1. I have sucessfully downloaded the NPU drivers from :
https://github.com/intel/linux-npu-driver/releases

2. I have an initialized VPU with :
dmesg | grep vpu
[ 2.001761] intel_vpu 0000:00:0b.0: enabling device (0000 -> 0002)
[ 2.014536] intel_vpu 0000:00:0b.0: [drm] Firmware: intel/vpu/vpu_37xx_v0.0.bin, version: 20240611MTL_CLIENT_SILICON-release0003ci_tag_ud202424_vpu_rc_20240611_0003f3e8a8f2747
[ 2.166758] [drm] Initialized intel_vpu 1.0.0 20230117 for 0000:00:0b.0 on minor 0

3. I also have an accel0 device passthrough working:
ls -l /dev/accel
total 0
crw-rw---- 1 root ssl-cert 261, 0 Aug 21 07:00 accel0

4. When I try to set the detector device to NPU in the Frigate config, I get the following message (see log below):
Cannot load library '/usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_npu_plugin.so': libze_loader.so.1: cannot open shared object file: No such file or directory

5. However, when looking inside the container, the plugin seems present:
ls -l /usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_npu_plugin.so
-rw-r--r-- 1 root root 1116121 Jul 23 14:43 /usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_npu_plugin.so

I observe the same behavior when using :

model:
path: /openvino-model/ssdlite_mobilenet_v2.xml

Has anyone tried the integrated Intel NPU with Openvino before ?

Many thanks,

Philippe

Version

0.14.0-da913d8

Frigate config file

mqtt:
  host: 192.168.5.252
  port: 1883
  user: xxxx
  password: xxxx
  topic_prefix: frigate

cameras:
  Cam_Garage: # <------ Name the camera
    ui:
      order: 3
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: 
            rtsp://xxxx:[email protected]:554/axis-media/media.amp?videocodec=h264&resolution=800x450&fps=5     
          roles:
            - detect
        - path: 
            rtsp://xxxx:[email protected]:554/axis-media/media.amp?videocodec=h264&resolution=1920x1080&fps=5 
          roles:
            - record
    zones:
      entire_garage:
        inertia: 3
        coordinates: 0,450,800,450,800,187,528,50,284,21,0,55
    detect:
      enabled: true
      width: 800
      height: 450 
      fps: 5
    motion:
      mask: 0.356,0.043,0.663,0.109,1,0.403,1,0,0,0,0,0.117
      improve_contrast: true
    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          threshold: 0.65 # 0.70 for Yolov8s_srt_100eV3. 0.6 for Yolov8s
          min_score: 0.55
          min_ratio: 0.20
          max_ratio: 0.8
    snapshots:
      required_zones:
        - entire_garage
      enabled: true
      bounding_box: true
      retain:
        default: 90
    record:
      enabled: true
      retain:
        days: 0
#       mode: active_objects
      events:
        retain:
          default: 30
          mode: active_objects
    review:
      alerts:
        required_zones:
          - entire_garage

  Cam_Garden: # <------ Name the camera
    ui:
      order: 1
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: 
            rtsp://xxxx:[email protected]:554/axis-media/media.amp?videocodec=h265&resolution=800x600&fps=5 
          roles:
            - detect
        - path: 
            rtsp://xxxx:[email protected]:554/axis-media/media.amp?videocodec=h265&resolution=1600x1200&fps=5 
          roles:
            - record
    zones:
      entire_garden:
        inertia: 3
        coordinates: 0,1,0.71,1,0.835,0.8,0.963,0.474,0.57,0.267,0.287,0.15,0,0.307
    detect:
      enabled: true  
      width: 800
      height: 600 
      fps: 5
    motion:
      mask: 0.29,0.141,0.58,0.264,0.695,0.322,1,0.485,1,0,0.278,0,0,0,0,0.296,0.162,0.212
      improve_contrast: true
      threshold: 30
      contour_area: 10
    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          threshold: 0.65 
          min_score: 0.55
          min_area: 250
          max_area: 10000
    snapshots:
      required_zones:
        - entire_garden
      enabled: true
      bounding_box: true
      retain:
        default: 90
    record:
      enabled: true
      retain:
        days: 0
#       mode: active_objects
      events:
        retain:
          default: 30
          mode: active_objects
    review:
      alerts:
        required_zones: entire_garden
  Cam_Cave: # <------ Name the camera
    ui:
      order: 2
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: 
            rtsp://xxxx:[email protected]:554/axis-media/media.amp?videocodec=h265&resolution=800x600&fps=5
          roles:
            - detect
        - path: 
            rtsp://xxxx:[email protected]:554/axis-media/media.amp?videocodec=h265&resolution=1600x1200&fps=5
          roles:
            - record
    zones:
      entire_cave:
        inertia: 3
        coordinates: 246,600,800,600,756,600,720,349,516,208,223,100,102,108,139,300,105,315,138,405
    detect:
      enabled: true
      width: 800 
      height: 600 
      fps: 5
    motion:
      mask:
        - 156,281,135,163,279,162,462,211,800,365,800,0,0,0,0,331
      improve_contrast: true
      threshold: 20
    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          threshold: 0.65 
          min_score: 0.55 
          min_area: 200
          max_area: 14000
          mask: 
            0.17,0.272,0.369,0.278,0.372,0.339,0.384,0.339,0.387,0.289,0.573,0.355,1,0.606,1,0,0,0.002,0,0.552,0.196,0.466
    snapshots:
      required_zones:
        - entire_cave
      enabled: true
      bounding_box: true
      retain:
        default: 90
    record:
      enabled: true
      retain:
        days: 0
#       mode: active_objects
      events:
        retain:
          default: 30
          mode: active_objects
    review:
      alerts:
        required_zones:
          - entire_cave

  Cam_Entree: # <------ Name the camera
    ui:
      order: 4
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: 
            rtsp://xxxx:[email protected]:554/axis-media/media.amp?videocodec=h264&resolution=640x480&fps=5 
          roles:
            - detect
        - path: 
            rtsp://xxxx:[email protected]:554/axis-media/media.amp?videocodec=h264&resolution=640x480&fps=5 
          roles:
            - record
    detect:
      enabled: true
      width: 640 
      height: 480 
      fps: 5
    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          threshold: 0.65
          min_score: 0.55
      mask: 1,0.357,1,0,0,0,0,0.321,0.593,0.325,0.601,0.346
    snapshots:
      enabled: true
      bounding_box: true
      retain:
        default: 90
    record:
      enabled: true
      retain:
        days: 0
#       mode: active_objects
      events:
        retain:
          default: 30
          mode: active_objects
    motion:
      mask: 0.001,0.002,0.999,0.001,0.999,0.355,0.602,0.346,0.592,0.322,0,0.32
      threshold: 40
      contour_area: 10
      improve_contrast: true

detectors:
  ov:
    type: openvino
    device: NPU
    model:
      path: /models/yolo_nas_m_256.onnx

model:
  model_type: yolonas
  width: 256 # <--- should match whatever was set in notebook
  height: 256 # <--- should match whatever was set in notebook
  input_tensor: nchw
  input_pixel_format: bgr
  labelmap_path: /models/coco_80cl.txt

version: 0.14
camera_groups: {}

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.14.0
    shm_size: "256mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/dri
      - /dev/accel
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /root/frigate/config/:/config/
      - /root/frigate:/media/frigate
      - /root/frigate/models:/models
      - type: tmpfs     # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "15000:5000"
      - "19350:1935" # RTMP feeds
    environment:
      PLUS_API_KEY: xxxx
      NEOReadDebugKeys: 1
      OverrideGpuAddressSpace: 48

Relevant log output

2024-08-21 09:28:04.483711629  Process detector:ov:
2024-08-21 09:28:04.484450278  Traceback (most recent call last):
2024-08-21 09:28:04.484477079    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-08-21 09:28:04.484479530      self.run()
2024-08-21 09:28:04.484480738    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-08-21 09:28:04.484485230      self._target(*self._args, **self._kwargs)
2024-08-21 09:28:04.484487045    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
2024-08-21 09:28:04.484500125      object_detector = LocalObjectDetector(detector_config=detector_config)
2024-08-21 09:28:04.484501349    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
2024-08-21 09:28:04.484502754      self.detect_api = create_detector(detector_config)
2024-08-21 09:28:04.484503632    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2024-08-21 09:28:04.484504447      return api(detector_config)
2024-08-21 09:28:04.484505400    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 43, in __init__
2024-08-21 09:28:04.484519659      self.interpreter = self.ov_core.compile_model(
2024-08-21 09:28:04.484520992    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 521, in compile_model
2024-08-21 09:28:04.484531392      super().compile_model(model, device_name, {} if config is None else config),
2024-08-21 09:28:04.484540486  RuntimeError: Exception from src/inference/src/cpp/core.cpp:123:
2024-08-21 09:28:04.484542622  Cannot load library '/usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_npu_plugin.so': libze_loader.so.1: cannot open shared object file: No such file or directory

Operating system

Proxmox

Install method

Docker Compose

Object Detector

OpenVino

Any other information that may be helpful

Asus NUC Intel Core Ultra 7 155H with iArc GPU.

philippedegeer · 2024-08-24T06:06:05Z

philippedegeer
Aug 24, 2024
Author

Bumping up this one....
Anyone having a view on whether this is a problem linked with my setup or something that Frigate needs to worked on ?
Many thanks - Philippe

0 replies

cheese202 · 2024-08-28T08:30:47Z

cheese202
Aug 28, 2024

Hi gyus. I have solved this issue installing the NPU drivers and from intel and updated OpenVino to the ltest version using pip and even once i got NPU working. Unfortunately I didn't get any screenshots to show how it works.
After installing drivers I got an error in Frigate log like "Can't find device". But after changing parameters in docker file of HA addon i gont it working. Iference time was about the 20ms and GPU was only utilised HW acceleration and CPU was 0%. But on the ARC GPU inference time was 2 times better something about 10ms.

CPU is Ultra 5 125H

I've installed all the drivers inside the docker container but didn't figure out how to commit changes and lauch it as the HA addon. So after container restart i just lost all progres, but for my opinion this would be great to add NPU support. It is really easy, working out of the box. And NPU can be really useful for some specific cases.

1 reply

philippedegeer Aug 29, 2024
Author

Thanks cheese202 for your comments. A least you got it to work. So my issue must be linked to my setup. I will go back to it next month and see what I can do.

azithromycin · 2024-10-10T05:27:53Z

azithromycin
Oct 10, 2024

I'm having the same issue. ASUS nuc14 pro with Intel Core Ultra 7 155H processor.

frigate log

2024-10-10 16:08:43.437226063 Process detector:ov:
2024-10-10 16:08:43.437972142 Traceback (most recent call last):
2024-10-10 16:08:43.437976187 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-10-10 16:08:43.437981779 self.run()
2024-10-10 16:08:43.437984675 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-10-10 16:08:43.437987203 self._target(*self._args, **self._kwargs)
2024-10-10 16:08:43.437988791 File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
2024-10-10 16:08:43.438011743 object_detector = LocalObjectDetector(detector_config=detector_config)
2024-10-10 16:08:43.438013843 File "/opt/frigate/frigate/object_detection.py", line 53, in init
2024-10-10 16:08:43.438015309 self.detect_api = create_detector(detector_config)
2024-10-10 16:08:43.438016649 File "/opt/frigate/frigate/detectors/init.py", line 18, in create_detector
2024-10-10 16:08:43.438038075 return api(detector_config)
2024-10-10 16:08:43.438039648 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 43, in init
2024-10-10 16:08:43.438041229 self.interpreter = self.ov_core.compile_model(
2024-10-10 16:08:43.438042713 File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 521, in compile_model
2024-10-10 16:08:43.438056400 super().compile_model(model, device_name, {} if config is None else config),
2024-10-10 16:08:43.438057857 RuntimeError: Exception from src/inference/src/cpp/core.cpp:123:
2024-10-10 16:08:43.438060287 Cannot load library '/usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_npu_plugin.so': libze_loader.so.1: cannot open shared object file: No such file or directory

frigate config

mqtt:
enabled: false

detectors: # <---- add detectors
#coral:
#type: edgetpu
#device: usb
ov:
type: openvino
device: NPU

model:
width: 300
height: 300
input_tensor: nhwc
input_pixel_format: bgr
path: /openvino-model/ssdlite_mobilenet_v2.xml
labelmap_path: /openvino-model/coco_91cl_bkgr.txt

go2rtc:
streams:
front_yard_camera:
- rtsp://frigate:[email protected]:554/Streaming/Channels/101 #Hikvision full stream 101
driveway_camera:
- rtsp://frigate:[email protected]:554/Streaming/Channels/101 #Hikvision full stream 101
doorbell_camera:
- ffmpeg:https://192.168.0.12/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=frigate&password=password#video=copy#audio=copy#audio=opus

cameras:
front_yard_camera: # <------ Name the camera
enabled: true
ffmpeg:
inputs:
- path: rtsp://frigate:[email protected]:554/Streaming/Channels/102 # <----- The stream you want to use for detection. hikvision substream 102
roles:
- detect
- path: rtsp://frigate:[email protected]:554/Streaming/Channels/101 # <----- high res stream for recording
roles:
- record
hwaccel_args: preset-vaapi
output_args:
record: preset-record-generic-audio-aac
detect:
enabled: true # <---- disable detection until you have a working camera feed
width: 1280
height: 720
motion:
mask: 0.022,0.055,0.316,0.052,0.315,0.097,0.02,0.099
zones: {}
review: {}
driveway_camera: # <------ Name the camera
enabled: true
ffmpeg:
inputs:
- path: rtsp://frigate:[email protected]:554/Streaming/Channels/102 # <----- The stream you want to use for detection. hikvision substream 102
roles:
- detect
- path: rtsp://frigate:[email protected]:554/Streaming/Channels/101 # <----- high res stream for recording
roles:
- record
hwaccel_args: preset-vaapi
output_args:
record: preset-record-generic-audio-aac
detect:
enabled: true # <---- disable detection until you have a working camera feed
width: 1280
height: 720
motion:
  mask: 0.021,0.054,0.02,0.096,0.317,0.099,0.318,0.054
doorbell_camera: # <------ Name the camera
enabled: true
ffmpeg:
inputs:
- path: rtsp://frigate:[email protected]:554/h265Preview_01_sub # <----- The stream you want to use for detection. hikvision substream 102
roles:
- detect
- path: rtsp://frigate:[email protected]:554/h265Preview_01_main # <----- high res stream for recording
roles:
- record
hwaccel_args: preset-vaapi
output_args:
record: preset-record-generic-audio-aac
detect:
enabled: true # <---- disable detection until you have a working camera feed
width: 1280
height: 720
zones: {}
motion:
mask: 0.011,0.015,0.307,0.017,0.306,0.049,0.009,0.048
review: {}

#test_video:
#ffmpeg:
#inputs:
#- path: /media/frigate/exports/front_yard_camera_nvmjyq.mp4
# input_args: -re -stream_loop -1 -fflags +genpts
# roles:
# - detect

record:
enabled: true
retain:
days: 14
mode: all
events:
retain:
default: 60
mode: motion

objects:
track:
- person
- cat
- dog
- bird

snapshots:
enabled: true
retain:
default: 30

version: 0.14

docker compose yaml

frigate:
container_name: frigate
privileged: true # this may not be necessary for all setups
restart: unless-stopped
image: ghcr.io/blakeblackshear/frigate:stable
shm_size: "256mb" # update for your cameras based on calculation above
devices:
#- /dev/bus/usb:/dev/bus/usb # Passes the USB Coral, needs to be modified for other versions
#- /dev/apex_0:/dev/apex_0 # Passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
#- /dev/video11:/dev/video11 # For Raspberry Pi 4B
- /dev/accel/accel0:/dev/accel/accel0
#- /dev/bus/usb:/dev/bus/usb # USB coral
- /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your hardware
volumes:
- /etc/localtime:/etc/localtime:ro
- /opt/frigate/config:/config
- /opt/frigate/storage:/media/frigate
- type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
target: /tmp/cache
tmpfs:
size: 1000000000
ports:
- "8971:8971"
#- "5000:5000" # Internal unauthenticated access. Expose carefully.
- "8554:8554" # RTSP feeds
- "8555:8555/tcp" # WebRTC over tcp
- "8555:8555/udp" # WebRTC over udp
environment:
#FRIGATE_RTSP_PASSWORD: "temp"
PLUS_API_KEY: "xxxxx"

I seem to have no problem accessing libopenvino_intel_npu_plugin.so from inside the container (docker exec)

root@2ceb3962bd53:/opt/frigate# ls -l /usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_npu_plugin.so
-rw-r--r-- 1 root root 1116121 Aug 25 23:03 /usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_npu_plugin.so
root@2ceb3962bd53:/opt/frigate#

Is anyone else having the same issue and has found a solution?

1 reply

NickM-27 Oct 10, 2024
Collaborator Sponsor

openvino has been updated for 0.15 so may work better there

cheese202 · 2024-10-11T07:40:33Z

cheese202
Oct 11, 2024

So i can confirm that intel NPU is works fine for me. There are the steps how i got it works.

used the latest dev build
installed NPU drivers to the host system according this guide and followed this guide completely including rights setup
added /dev/accel/accel0 to the dockerfile
Installed the same inside the docker container, restarted it and everything works

So i'm not a pro in github usage but i'll try to create the pull request to add intel NPU drivers in dev container because for this you just need to install one package from debial bookwork and 4 packages from intel.

The main 2 problems which i faced with are:

I'm using home assistant addon for this and i'm unable to commit changes to the container then run it. So if i will reboot container through the HA UI i will loose all installed data.
I'm unable to run custom trained onnx yolonas model on NPU. Only integrated model in frigate can run on it

also it seems that detector CPU load shows incorrect data because htop shows host load no more than 18%

So it would be awesome to add NPU support because it can be very potential for some cases and also feature mentioned somewhere there to add the possibility to use separate detector for specific camera

2 replies

NickM-27 Oct 11, 2024
Collaborator Sponsor

also it seems that detector CPU load shows incorrect data because htop shows host load no more than 18%

It's showing correct data, it's calculated inside the container as the % of a single core

philippedegeer Nov 13, 2024
Author

Thanks @cheese202, I managed to get the NPU work using your last post. Like you I can run the integrated ssdlite model, but not the yolonas models (frigate+ trained or COCO trained).
I am not an AI expert but may be this is the issue :

https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html#limitations

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Detector Support]: Openvino and Intel integrated NPU support #13248

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[Detector Support]: Openvino and Intel integrated NPU support #13248

philippedegeer Aug 21, 2024

Describe the problem you are having

Version

Frigate config file

docker-compose file or Docker CLI command

Relevant log output

Operating system

Install method

Object Detector

Any other information that may be helpful

Replies: 4 comments · 4 replies

philippedegeer Aug 24, 2024 Author

cheese202 Aug 28, 2024

philippedegeer Aug 29, 2024 Author

azithromycin Oct 10, 2024

NickM-27 Oct 10, 2024 Collaborator Sponsor

cheese202 Oct 11, 2024

NickM-27 Oct 11, 2024 Collaborator Sponsor

philippedegeer Nov 13, 2024 Author

philippedegeer
Aug 21, 2024

Replies: 4 comments 4 replies

philippedegeer
Aug 24, 2024
Author

cheese202
Aug 28, 2024

philippedegeer Aug 29, 2024
Author

azithromycin
Oct 10, 2024

NickM-27 Oct 10, 2024
Collaborator Sponsor

cheese202
Oct 11, 2024

NickM-27 Oct 11, 2024
Collaborator Sponsor

philippedegeer Nov 13, 2024
Author