[Detector Support]: OpenVino crash when using GPU #13388

jjak0b · 2024-08-27T21:54:10Z

jjak0b
Aug 27, 2024

Describe the problem you are having

I have a Proxmox 8.2.4 installation running on an Intel N3350 CPUì and a LXC unprivileged Debian 12 container running Dcoker which runs a Frigate Container.

The issue is that OpenVino with GPU detection crash the Frigate container but if i set CPU in detector type won't crash.
Hardware acceleration works and is detected using intel_gpu_top in proxmox host. (drivers are already installed)

The Frigate configuration i provide in this issue is about a UVC camera passed to Frigate with USB passthrough, but the same issue happens also using an IP Camera.

--
Proxmox 8.2.4 with kernel 6.8.12-1-pve
Docker installed in LXC: 27.1.1

Version

0.14.0

Frigate config file

database:
  path: /db/frigate.db

mqtt:
  enabled: true
  host: 192.168.1.200
  port: 1883
  user: '{FRIGATE_MQTT_USER}'
  password: '{FRIGATE_MQTT_PASSWORD}'
  topic_prefix: frigate
  client_id: frigate

telemetry:
  # Optional: Configure system stats
  stats:
    amd_gpu_stats: false
    intel_gpu_stats: true
    network_bandwidth: true

go2rtc:
  rtsp:
    username: '{FRIGATE_MQTT_USER}'
    password: '{FRIGATE_MQTT_PASSWORD}'
  streams:
    uvc_camera: ffmpeg:device?video=0&video_size=640x480#video=h264#hardware=vaapi#raw=-pix_fmt yuv420p
    leovin_cam:
      - rtsp://admin:[email protected]/stream0
      - ffmpeg:leovin_cam#video=copy#audio=aac#hardware
  webrtc:
    candidates:
      - 192.168.1.202:8555
      - stun:8555
birdseye:
  enabled: true
  mode: motion

# works
detectors:
  ov:
    type: openvino
    device: GPU

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  path: /openvino-model/ssdlite_mobilenet_v2.xml
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

record:
  enabled: true
  events:
    retain:
      default: 5
      mode: active_objects
    pre_capture: 5
    post_capture: 20

objects:
  track:
    - person
    - bicycle
    - motorcycle
  filters:
    person:
      threshold: 0.65
    bicycle:
      threshold: 0.65
    motorcycle:
      threshold: 0.65
    airplane:
      threshold: 0.70

snapshots:
  enabled: true
  timestamp: true
  bounding_box: true
  retain:
    default: 5

ffmpeg:
  hwaccel_args: preset-vaapi

cameras:

  Retro_UVC_Computer:
    enabled: true
    live:
      stream_name: uvc_camera
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/uvc_camera
          roles:
            - detect
            - record
      input_args: preset-rtsp-restream

    zones:
      external_door:
        coordinates: 640,480,105,480,113,0,640,0
        objects:
          - person
    detect:
      enabled: true
    record:
      enabled: true
    snapshots:
      enabled: true

version: 0.14

docker-compose file or Docker CLI command

docker run -d \
  --name frigateT \
  --restart=unless-stopped \
  --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \
  --device /dev/video0:/dev/video0 \
  --device /dev/dri:/dev/dri \
  --network=bridge \
  --privileged \
  --cap-add=CAP_PERFMON \
  --group-add video \
  --group-add "106" \
  --group-add "44" \
  --shm-size=512m \
  --device-cgroup-rule="c 189:* rmw" \
  --device-cgroup-rule="c 226:* rmw" \
  -v /mnt/NVR:/media:rw \
  -v /portainer/Files/AppData/Config/FrigateT:/config:rw \
  -v /etc/localtime:/etc/localtime:ro \
  -v /portainer/Files/AppData/Data/FrigateT/db:/db:rw \
  -e LIBVA_DRIVER_NAME=i965 \
  -e FRIGATE_MQTT_USER=user\
  -e FRIGATE_MQTT_PASSWORD=mqttpass\
  -e FRIGATE_RTSP_PASSWORD='rtsppass' \
  -e TZ=Europe/Rome \
  -p 5000:5000 \
  -p 8971:8971 \
  -p 8554:8554 \
  -p 8555:8555/tcp \
  -p 8555:8555/udp \
  ghcr.io/blakeblackshear/frigate:0.14.0

Relevant Frigate log output

Process detector:ov:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
    object_detector = LocalObjectDetector(detector_config=detector_config)
  File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
    self.detect_api = create_detector(detector_config)
  File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
    return api(detector_config)
  File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 43, in __init__
    self.interpreter = self.ov_core.compile_model(
  File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 521, in compile_model
    super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: Exception from src/inference/src/cpp/core.cpp:123:
Exception from src/inference/src/dev/plugin.cpp:59:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:178:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:433:
[GPU] clWaitForEvents, error code: -14

Relevant go2rtc log output

...

[2024-08-27 23:16:03] peewee_migrate.logs            INFO    : Starting migrations
[2024-08-27 23:16:03] peewee_migrate.logs            INFO    : There is nothing to migrate
[2024-08-27 23:16:03] frigate.app                    INFO    : Recording process started: 286
[2024-08-27 23:16:03] frigate.app                    INFO    : Recording process started: 288
[2024-08-27 23:16:03] frigate.app                    INFO    : go2rtc process pid: 99
[2024-08-27 23:16:03] detector.ov                    INFO    : Starting detection process: 315
[2024-08-27 23:16:03] frigate.app                    INFO    : Output process started: 317
[2024-08-27 23:16:03] frigate.app                    INFO    : Camera processor started for Retro_UVC_Computer: 332
[2024-08-27 23:16:03] frigate.app                    INFO    : Capture process started for Retro_UVC_Computer: 335
[INFO] Starting go2rtc healthcheck service...
[2024-08-27 23:16:07] frigate.util.services          ERROR   : Unable to poll intel GPU stats: Failed to initialize PMU! (Permission denied)
Process detector:ov:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
    object_detector = LocalObjectDetector(detector_config=detector_config)
  File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
    self.detect_api = create_detector(detector_config)
  File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
    return api(detector_config)
  File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 43, in __init__
    self.interpreter = self.ov_core.compile_model(
  File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 521, in compile_model
    super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: Exception from src/inference/src/cpp/core.cpp:123:
Exception from src/inference/src/dev/plugin.cpp:59:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:178:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:433:
[GPU] clWaitForEvents, error code: -14

[2024-08-27 23:16:24] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting Frigate...

...

Operating system

Proxmox

Install method

Docker CLI

Object Detector

OpenVino

Screenshots of the Frigate UI's System metrics pages

N/A

Any other information that may be helpful

Proxmox unprivileged container lxc.conf

arch: amd64
cores: 2
features: keyctl=1,nesting=1
hostname: dockero
memory: 5120
mp0: filedata:100/vm-100-disk-1.raw,mp=/portainer,size=9G
mp1: /mnt/net_shared/Videos,mp=/mnt/Videos
mp2: /mnt/net_shared/NVR,mp=/mnt/NVR
mp3: /mnt/net_shared/Music,mp=/mnt/Music
mp5: /mnt/net_games,mp=/mnt/Games
nameserver: 192.168.1.254
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.254,hwaddr=E2:2C:70:45:F5:A2,ip=192.168.1.202/24,type=veth
onboot: 1
ostype: debian
rootfs: filedata:100/vm-100-disk-0.raw,size=21G
startup: up=1
swap: 1024
unprivileged: 1
lxc.cgroup.memory.swappiness: 1
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 189:* rwm
lxc.cgroup2.devices.allow: c 81:* rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/bus/usb/001/002 dev/bus/usb/001/002 none bind,optional,create=file
lxc.mount.entry: /dev/video0 dev/video0 none bind,optional,create=file
lxc.mount.entry: /dev/video1 dev/video1 none bind,optional,create=file
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 44
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 61
lxc.idmap: g 106 104 1
lxc.idmap: g 107 100107 65429

hawkeye217 · 2024-08-27T21:57:22Z

hawkeye217
Aug 27, 2024
Collaborator

See the FAQ, Upgrade issues question 6.

2 replies

jjak0b Aug 27, 2024
Author

Sorry i missed it. But i specify also that Frigate didn't work with GPU Detection also In 0.13 probably also caused by #12266

NickM-27 Aug 27, 2024
Collaborator Sponsor

have you tried the fixes listed there?

Jonoox · 2024-09-09T16:28:35Z

Jonoox
Sep 9, 2024

So what can we do about this? I'm on Pentium J5005 and having same problem with GPU on OpenVino and frigate 14.1

6 replies

Jonoox Sep 11, 2024

I have weaker CPU and I'm getting 80-160ms of inference time.

Jonoox Sep 12, 2024

With ssdmobilenetv2, two cameras and CPU detector on J5005 I have bad inference speed, going to over 300ms sometimes when eg. car headlights makes motion detector go crazy. I would really need to get GPU detector working.

Jonoox Oct 18, 2024

I upgraded the host from Intel J5005 terminal to Intel N100 chinese box and everything works with frigate almost without any special configurations. Openvino GPU acceleration and 15-20ms inference speed with barely any CPU usage.

ivanjx Oct 19, 2024

@Jonoox do you use hardware acceleration too?

Jonoox Oct 19, 2024

Yes, vaapi preset. My frigate is running on proxmox, installed by tteck script.
vainfo: VA-API version: 1.17 (libva 2.10.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.1.1 ()

giuliodali · 2024-11-20T23:04:47Z

giuliodali
Nov 20, 2024

Hi, I'm running Frigate 0.15-beta2 on an Intel J4125 as an add-on to Home Assistant OS

Core              2024.11.2
Supervisor        2024.11.2
Operating System  13.2
Frontend          20241106.2

with frigate config:

detectors:
  ov_0:
    type: openvino
    device: GPU
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

I get this in the logs too:

2024-11-20 22:40:51.631246947  Process detector:ov_0:
2024-11-20 22:40:51.631253833  Traceback (most recent call last):
2024-11-20 22:40:51.631255680    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-11-20 22:40:51.631262348      self.run()
2024-11-20 22:40:51.631264933    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
2024-11-20 22:40:51.631266673      return run(*args, **kwargs)
2024-11-20 22:40:51.631313014    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-11-20 22:40:51.631314993      self._target(*self._args, **self._kwargs)
2024-11-20 22:40:51.631317433    File "/opt/frigate/frigate/object_detection.py", line 121, in run_detector
2024-11-20 22:40:51.631319102      object_detector = LocalObjectDetector(detector_config=detector_config)
2024-11-20 22:40:51.631320869    File "/opt/frigate/frigate/object_detection.py", line 68, in __init__
2024-11-20 22:40:51.631322325      self.detect_api = create_detector(detector_config)
2024-11-20 22:40:51.631363185    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2024-11-20 22:40:51.631364958      return api(detector_config)
2024-11-20 22:40:51.631387101    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 40, in __init__
2024-11-20 22:40:51.631389073      self.interpreter = self.ov_core.compile_model(
2024-11-20 22:40:51.631391326    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 543, in compile_model
2024-11-20 22:40:51.631392964      super().compile_model(model, device_name, {} if config is None else config),
2024-11-20 22:40:51.631394571  RuntimeError: Exception from src/inference/src/cpp/core.cpp:121:
2024-11-20 22:40:51.631395986  Exception from src/inference/src/dev/plugin.cpp:53:
2024-11-20 22:40:51.631397611  Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:185:
2024-11-20 22:40:51.631421102  [GPU] ProgramBuilder build failed!
2024-11-20 22:40:51.631423253  Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:429:
2024-11-20 22:40:51.631424646  [GPU] clWaitForEvents, error code: -14

I've found it discussed here and here on the intel boards, discussing Frigate.

The suggestions are:

Ensure frigate is on the latest intel/compute-runtime 24.39.31294.12, which is newer than what was recently pinned
From here setting i915.enable_hangcheck=0 i915.request_timeout_ms=100000 in grub as per docs.

Can these be tried?

FYI - @NickM-27

4 replies

hawkeye217 Nov 21, 2024
Collaborator

There are several reasons why we pinned the Intel compute-runtime, it may not be possible to unpin it at the moment.

You'll want to read the docs for properly configuring the OpenVINO detector, by the way. Your config file is incorrect: https://deploy-preview-13787--frigate-docs.netlify.app/configuration/object_detectors/#ssdlite-mobilenet-v2

NickM-27 Nov 21, 2024
Collaborator Sponsor

To be clear Celeron is not officially supported for openvino

giuliodali Nov 22, 2024

According to this copy of the docs, as long as the GNA plugin is enabled then the Celeron J4125 would work?

(Config above fixed)

NickM-27 Nov 23, 2024
Collaborator Sponsor

GNA does not support running object detection models

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Detector Support]: OpenVino crash when using GPU #13388

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 12 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

[Detector Support]: OpenVino crash when using GPU #13388

Describe the problem you are having

Version

Frigate config file

docker-compose file or Docker CLI command

Relevant Frigate log output

Relevant go2rtc log output

Operating system

Install method

Object Detector

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Replies: 3 comments · 12 replies

hawkeye217 Aug 27, 2024 Collaborator

jjak0b Aug 27, 2024 Author

NickM-27 Aug 27, 2024 Collaborator Sponsor

hawkeye217 Nov 21, 2024 Collaborator

NickM-27 Nov 21, 2024 Collaborator Sponsor

NickM-27 Nov 23, 2024 Collaborator Sponsor

Replies: 3 comments 12 replies

hawkeye217
Aug 27, 2024
Collaborator

jjak0b Aug 27, 2024
Author

NickM-27 Aug 27, 2024
Collaborator Sponsor

hawkeye217 Nov 21, 2024
Collaborator

NickM-27 Nov 21, 2024
Collaborator Sponsor

NickM-27 Nov 23, 2024
Collaborator Sponsor