[HW Accel Support]: After enabling iGPU passthrough, lots of errors that appear to kill and restart threads

[biggt]

Describe the problem you are having

I recently enabled iGPU passthrough for the Proxmox VM; Ever since, I keep getting error messages indicating that all the cameras fail to read the frames from ffmpeg process, it fails to sync surface, fails to download from, and has an Input/output error.

HP ProDesk 400 G4 (Intel i5-8500t 6C6T, 32GB RAM)
  Proxmox 8.2.7 - Linux 6.8.12-3-pve
    Ubuntu 24.04.1 VM (5 cores)
      Docker Compose
        Frigate 0.15.0-33825f6

However, since enabling the iGPU passthrough, I have noticed that the General Metrics page has been slower than normal, which may be a separate issue.

Version

0.15.0-33825f6

Frigate config file

database:
  path: /config/frigate.db

mqtt:
  enabled: true
  host: 10.0.102.100
  port: 1883
  topic_prefix: frigate
  user: mqtt
  password: {password}
  stats_interval: 60

detectors:
  coral:
    type: edgetpu
    device: usb


record:
  sync_recordings: true
  enabled: true
  expire_interval: 60 ## Minutes, not days.
  retain:
    days: 60
    mode: all
  alerts:
    retain:
      days: 60
      mode: motion
    pre_capture: 5
    post_capture: 15

  detections:
    retain:
      days: 60
      mode: motion
    pre_capture: 5
    post_capture: 15


ffmpeg:
  hwaccel_args: preset-vaapi # Use Intel NUC iGPU (optional)
  # hwaccel_args: preset-intel-qsv-h265 # Use Intel NUC iGPU (optional) ## Non-Functional for all cameras
  # hwaccel_args: preset-intel-qsv-h264 # Use Intel NUC iGPU (optional) ## Non-Functional for all cameras
  output_args:
    # Optional: output args for detect streams (default: shown below)
    # detect: -threads 2 -f rawvideo -pix_fmt yuv420p -filter:v fps=fps=5 ## appears to be borken for all cameras... "could not open encoder before EOF"
    # detect: -threads 2 ## appears to be borken for all cameras... "could not open encoder before EOF"
    record: preset-record-generic-audio-aac
  retry_interval: 10


objects:
  track:
    - person
    - dog
    - cat
    - car
    - horse
    - bicycle
    - motorcycle
    #- boat
    #- hat
    - backpack
    - license_plate
  filters:
    person:
      min_area: 5000
      max_area: 100000
      min_ratio: 0.5
      max_ratio: 2.0
      min_score: 0.5
      threshold: 0.7

detect:
  stationary:
    interval: 60
    threshold: 60


go2rtc:
  webrtc:
    candidates:
      - 10.0.2.1:8555
      - stun:8555
      - 127.0.0.1:8555
  streams:
    Amcrest_DiningRoom_Cages_go2rtc_main:
      - ffmpeg:rtsp://frigate:{password}@10.0.1.130:554/cam/realmonitor?channel=1&subtype=0#video=copy#audio=copy#audio=aac
    Amcrest_DiningRoom_Cages_go2rtc_sub:
      - ffmpeg:rtsp://frigate:{password}@10.0.1.130:554/cam/realmonitor?channel=1&subtype=1#video=copy

    Amcrest_DiningRoom_Hallway_go2rtc_main:
      - ffmpeg:rtsp://frigate:{password}@10.0.1.131:554/cam/realmonitor?channel=1&subtype=0#video=copy#audio=copy#audio=aac
    Amcrest_DiningRoom_Hallway_go2rtc_sub:
      - ffmpeg:rtsp://frigate:{password}@10.0.1.131:554/cam/realmonitor?channel=1&subtype=1#video=copy

    Reolink_Video_Doorbell_go2rtc_main:
      - ffmpeg:https://10.0.1.120/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=frigate&password={password}#video=copy#audio=copy#audio=opus
    Reolink_Video_Doorbell_go2rtc_sub:
      - ffmpeg:https://10.0.1.120/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=frigate&password={password}#video=copy



cameras:
  Cages:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Amcrest_DiningRoom_Cages_go2rtc_main
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/Amcrest_DiningRoom_Cages_go2rtc_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    live:
      stream_name: Amcrest_DiningRoom_Cages_go2rtc_main

  Hallway:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Amcrest_DiningRoom_Hallway_go2rtc_main
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/Amcrest_DiningRoom_Hallway_go2rtc_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    live:
      stream_name: Amcrest_DiningRoom_Hallway_go2rtc_main



  reolink_video_doorbell_camera:
    live:
      stream_name: Reolink_Video_Doorbell_go2rtc_main
    zones:
      zone_front_porch_steps:
        coordinates: 469,140,483,672,219,672,207,172
      zone_front_porch_walkway:
        coordinates: 469,672,523,672,530,625,896,618,896,372,485,453
    motion:
      mask:
        - 208,336,896,301,896,0,0,0,0,672,215,672 # Left-side, ceiling, right-side above railing blocked  
    ffmpeg:

      inputs:
        - path: rtsp://127.0.0.1:8554/Reolink_Video_Doorbell_go2rtc_main
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/Reolink_Video_Doorbell_go2rtc_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    review:
      alerts:
        required_zones:
          - zone_front_porch_steps
          - zone_front_porch_walkway

ui:
  timezone: America/New_York

  time_format: browser
  date_style: medium
  time_style: medium

  strftime_fmt: '%Y/%m/%d %H:%M'

telemetry:
  version_check: true
version: 0.15-0
camera_groups:
  Dining_Room:
    order: 1
    icon: LuApple
    cameras:
      - Cages
      - Hallway

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.15.0-beta1
    shm_size: "256mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
#      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      #- ./config/config.yml:/config/config.yml
      - ./config/:/config/
      - /Frigate/frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "{password}"
      TZ: America/New_York

Relevant Frigate log output

2024-11-21 14:59:36.848117779  [INFO] Preparing Frigate...
2024-11-21 14:59:36.856146912  [INFO] Starting Frigate...
2024-11-21 14:59:41.155849456  [2024-11-21 09:59:41] frigate.util.config            INFO    : Checking if frigate config needs migration...
2024-11-21 14:59:41.231060084  [2024-11-21 09:59:41] frigate.util.config            INFO    : frigate config does not need migration...
2024-11-21 15:00:02.259165785  [2024-11-21 10:00:02] frigate.app                    INFO    : Starting Frigate (0.15.0-33825f6)
2024-11-21 15:00:02.281403371  [2024-11-21 10:00:02] peewee_migrate.logs            INFO    : Starting migrations
2024-11-21 15:00:02.333531637  [2024-11-21 10:00:02] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-11-21 15:00:02.352471539  [2024-11-21 10:00:02] frigate.app                    INFO    : Recording process started: 375
2024-11-21 15:00:02.352710198  [2024-11-21 10:00:02] frigate.app                    INFO    : Review process started: 376
2024-11-21 15:00:02.361975982  [2024-11-21 10:00:02] frigate.app                    INFO    : go2rtc process pid: 98
2024-11-21 15:00:02.391401175  [2024-11-21 10:00:02] detector.coral                 INFO    : Starting detection process: 392
2024-11-21 15:00:02.408394767  [2024-11-21 10:00:02] frigate.app                    INFO    : Output process started: 411
2024-11-21 15:00:03.060668549  [2024-11-21 10:00:03] frigate.app                    INFO    : Camera processor started for Cages: 453
2024-11-21 15:00:03.072123882  [2024-11-21 10:00:03] frigate.app                    INFO    : Camera processor started for Hallway: 456
2024-11-21 15:00:03.085723921  [2024-11-21 10:00:03] frigate.app                    INFO    : Camera processor started for reolink_video_doorbell_camera: 464
2024-11-21 15:00:03.100117399  [2024-11-21 10:00:03] frigate.app                    INFO    : Capture process started for Cages: 472
2024-11-21 15:00:03.110570006  [2024-11-21 10:00:03] frigate.app                    INFO    : Capture process started for Hallway: 480
2024-11-21 15:00:03.116371260  [2024-11-21 10:00:03] frigate.app                    INFO    : Capture process started for reolink_video_doorbell_camera: 484
2024-11-21 15:00:04.158950443  [2024-11-21 10:00:04] frigate.api.fastapi_app        INFO    : Starting FastAPI app
2024-11-21 15:00:04.361241056  [2024-11-21 10:00:04] frigate.api.fastapi_app        INFO    : FastAPI started
2024-11-21 15:00:05.014765181  [2024-11-21 10:00:02] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-11-21 15:00:05.022772394  [2024-11-21 10:00:05] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-11-21 15:02:31.562425603  [2024-11-21 10:02:31] frigate.video                  ERROR   : Cages: Unable to read frames from ffmpeg process.
2024-11-21 15:02:31.562542440  [2024-11-21 10:02:31] frigate.video                  ERROR   : Cages: ffmpeg process is not running. exiting capture thread...
2024-11-21 15:02:33.165992253  [2024-11-21 10:02:33] watchdog.Cages                 ERROR   : Ffmpeg process crashed unexpectedly for Cages.
2024-11-21 15:02:33.166093366  [2024-11-21 10:02:33] watchdog.Cages                 ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-11-21 15:02:33.166654650  [2024-11-21 10:02:33] ffmpeg.Cages.detect            ERROR   : [AVHWFramesContext @ 0x772600052d40] Failed to sync surface 0xb: 1 (operation failed).
2024-11-21 15:02:33.166924845  [2024-11-21 10:02:33] ffmpeg.Cages.detect            ERROR   : [hwdownload @ 0x7725fc003980] Failed to download frame: -5.
2024-11-21 15:02:33.166958627  [2024-11-21 10:02:33] ffmpeg.Cages.detect            ERROR   : [vf#0:0 @ 0x5ca831f0a200] Error while filtering: Input/output error
2024-11-21 15:02:33.167376270  [2024-11-21 10:02:33] ffmpeg.Cages.detect            ERROR   : [vf#0:0 @ 0x5ca831f0a200] Task finished with error code: -5 (Input/output error)
2024-11-21 15:02:33.167681731  [2024-11-21 10:02:33] ffmpeg.Cages.detect            ERROR   : [vf#0:0 @ 0x5ca831f0a200] Terminating thread with return code -5 (Input/output error)
2024-11-21 15:02:35.181565553  [2024-11-21 10:02:35] frigate.video                  ERROR   : Hallway: Unable to read frames from ffmpeg process.
2024-11-21 15:02:35.181964352  [2024-11-21 10:02:35] frigate.video                  ERROR   : Hallway: ffmpeg process is not running. exiting capture thread...
2024-11-21 15:02:43.191245956  [2024-11-21 10:02:43] watchdog.Hallway               ERROR   : Ffmpeg process crashed unexpectedly for Hallway.
2024-11-21 15:02:43.191251174  [2024-11-21 10:02:43] watchdog.Hallway               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-11-21 15:02:43.191253920  [2024-11-21 10:02:43] ffmpeg.Hallway.detect          ERROR   : [AVHWFramesContext @ 0x706c90053740] Failed to sync surface 0x13: 1 (operation failed).
2024-11-21 15:02:43.191259472  [2024-11-21 10:02:43] ffmpeg.Hallway.detect          ERROR   : [hwdownload @ 0x706c88003980] Failed to download frame: -5.
2024-11-21 15:02:43.191285728  [2024-11-21 10:02:43] ffmpeg.Hallway.detect          ERROR   : [vf#0:0 @ 0x55c7f0117c80] Error while filtering: Input/output error
2024-11-21 15:02:43.191288585  [2024-11-21 10:02:43] ffmpeg.Hallway.detect          ERROR   : [vf#0:0 @ 0x55c7f0117c80] Task finished with error code: -5 (Input/output error)
2024-11-21 15:02:43.191321035  [2024-11-21 10:02:43] ffmpeg.Hallway.detect          ERROR   : [vf#0:0 @ 0x55c7f0117c80] Terminating thread with return code -5 (Input/output error)
2024-11-21 15:05:04.670334428  [2024-11-21 10:05:04] frigate.video                  ERROR   : Cages: Unable to read frames from ffmpeg process.
2024-11-21 15:05:04.670392081  [2024-11-21 10:05:04] frigate.video                  ERROR   : Cages: ffmpeg process is not running. exiting capture thread...
2024-11-21 15:05:12.095038540  [2024-11-21 10:05:12] frigate.video                  ERROR   : Hallway: Unable to read frames from ffmpeg process.
2024-11-21 15:05:12.095111031  [2024-11-21 10:05:12] frigate.video                  ERROR   : Hallway: ffmpeg process is not running. exiting capture thread...
2024-11-21 15:05:13.199848805  [2024-11-21 10:05:13] watchdog.Cages                 ERROR   : Ffmpeg process crashed unexpectedly for Cages.
2024-11-21 15:05:13.199882508  [2024-11-21 10:05:13] watchdog.Cages                 ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-11-21 15:05:13.200054738  [2024-11-21 10:05:13] ffmpeg.Cages.detect            ERROR   : [AVHWFramesContext @ 0x71cf48054d80] Failed to sync surface 0xf: 1 (operation failed).
2024-11-21 15:05:13.200082663  [2024-11-21 10:05:13] ffmpeg.Cages.detect            ERROR   : [hwdownload @ 0x71cf58003980] Failed to download frame: -5.
2024-11-21 15:05:13.200357874  [2024-11-21 10:05:13] ffmpeg.Cages.detect            ERROR   : [vf#0:0 @ 0x5d89ade0c340] Error while filtering: Input/output error
2024-11-21 15:05:13.200362088  [2024-11-21 10:05:13] ffmpeg.Cages.detect            ERROR   : [vf#0:0 @ 0x5d89ade0c340] Task finished with error code: -5 (Input/output error)
2024-11-21 15:05:13.200364970  [2024-11-21 10:05:13] ffmpeg.Cages.detect            ERROR   : [vf#0:0 @ 0x5d89ade0c340] Terminating thread with return code -5 (Input/output error)
2024-11-21 15:05:13.221151573  [2024-11-21 10:05:13] watchdog.Hallway               ERROR   : Ffmpeg process crashed unexpectedly for Hallway.
2024-11-21 15:05:13.221156885  [2024-11-21 10:05:13] watchdog.Hallway               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-11-21 15:05:13.221159662  [2024-11-21 10:05:13] ffmpeg.Hallway.detect          ERROR   : [AVHWFramesContext @ 0x73d964053840] Failed to sync surface 0x12: 1 (operation failed).
2024-11-21 15:05:13.221162139  [2024-11-21 10:05:13] ffmpeg.Hallway.detect          ERROR   : [hwdownload @ 0x73d974003980] Failed to download frame: -5.
2024-11-21 15:05:13.222066093  [2024-11-21 10:05:13] ffmpeg.Hallway.detect          ERROR   : [vf#0:0 @ 0x5c7b71378ec0] Error while filtering: Input/output error
2024-11-21 15:05:13.222865853  [2024-11-21 10:05:13] ffmpeg.Hallway.detect          ERROR   : [vf#0:0 @ 0x5c7b71378ec0] Task finished with error code: -5 (Input/output error)
2024-11-21 15:05:13.222869788  [2024-11-21 10:05:13] ffmpeg.Hallway.detect          ERROR   : [vf#0:0 @ 0x5c7b71378ec0] Terminating thread with return code -5 (Input/output error)
2024-11-21 15:07:35.507768423  [2024-11-21 10:07:35] frigate.video                  ERROR   : Hallway: Unable to read frames from ffmpeg process.
2024-11-21 15:07:35.509143469  [2024-11-21 10:07:35] frigate.video                  ERROR   : Hallway: ffmpeg process is not running. exiting capture thread...
2024-11-21 15:07:43.251115080  [2024-11-21 10:07:43] watchdog.Hallway               ERROR   : Ffmpeg process crashed unexpectedly for Hallway.
2024-11-21 15:07:43.251150341  [2024-11-21 10:07:43] watchdog.Hallway               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-11-21 15:07:43.251359637  [2024-11-21 10:07:43] ffmpeg.Hallway.detect          ERROR   : [AVHWFramesContext @ 0x755e00052400] Failed to sync surface 0x14: 1 (operation failed).
2024-11-21 15:07:43.251363813  [2024-11-21 10:07:43] ffmpeg.Hallway.detect          ERROR   : [hwdownload @ 0x755e10003980] Failed to download frame: -5.
2024-11-21 15:07:43.251889081  [2024-11-21 10:07:43] ffmpeg.Hallway.detect          ERROR   : [vf#0:0 @ 0x5eedf309d2c0] Error while filtering: Input/output error
2024-11-21 15:07:43.252505766  [2024-11-21 10:07:43] ffmpeg.Hallway.detect          ERROR   : [vf#0:0 @ 0x5eedf309d2c0] Task finished with error code: -5 (Input/output error)
2024-11-21 15:07:43.252535804  [2024-11-21 10:07:43] ffmpeg.Hallway.detect          ERROR   : [vf#0:0 @ 0x5eedf309d2c0] Terminating thread with return code -5 (Input/output error)
2024-11-21 15:07:47.865143050  [2024-11-21 10:07:47] frigate.video                  ERROR   : Cages: Unable to read frames from ffmpeg process.
2024-11-21 15:07:47.865516281  [2024-11-21 10:07:47] frigate.video                  ERROR   : Cages: ffmpeg process is not running. exiting capture thread...
2024-11-21 15:07:53.232037980  [2024-11-21 10:07:53] watchdog.Cages                 ERROR   : Ffmpeg process crashed unexpectedly for Cages.
2024-11-21 15:07:53.232479983  [2024-11-21 10:07:53] watchdog.Cages                 ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-11-21 15:07:53.232484422  [2024-11-21 10:07:53] ffmpeg.Cages.detect            ERROR   : [AVHWFramesContext @ 0x7e9598053000] Failed to sync surface 0x13: 1 (operation failed).
2024-11-21 15:07:53.232687083  [2024-11-21 10:07:53] ffmpeg.Cages.detect            ERROR   : [hwdownload @ 0x7e95a4003980] Failed to download frame: -5.
2024-11-21 15:07:53.232901212  [2024-11-21 10:07:53] ffmpeg.Cages.detect            ERROR   : [vf#0:0 @ 0x5e1047c20ac0] Error while filtering: Input/output error
2024-11-21 15:07:53.232904716  [2024-11-21 10:07:53] ffmpeg.Cages.detect            ERROR   : [vf#0:0 @ 0x5e1047c20ac0] Task finished with error code: -5 (Input/output error)
2024-11-21 15:07:53.232956480  [2024-11-21 10:07:53] ffmpeg.Cages.detect            ERROR   : [vf#0:0 @ 0x5e1047c20ac0] Terminating thread with return code -5 (Input/output error)

Relevant go2rtc log output

2024-11-21 14:59:36.847329585  [INFO] Preparing new go2rtc config...
2024-11-21 14:59:37.364857012  [INFO] Not injecting WebRTC candidates into go2rtc config as it has been set manually
2024-11-21 14:59:37.411053769  [INFO] Starting go2rtc...
2024-11-21 14:59:37.597424761  09:59:37.597 INF go2rtc platform=linux/amd64 revision=b2399f3 version=1.9.2
2024-11-21 14:59:37.597429185  09:59:37.597 INF config path=/dev/shm/go2rtc.yaml
2024-11-21 14:59:37.598462840  09:59:37.598 INF [api] listen addr=:1984
2024-11-21 14:59:37.598466307  09:59:37.598 INF [rtsp] listen addr=:8554
2024-11-21 14:59:37.599123551  09:59:37.599 INF [webrtc] listen addr=:8555
2024-11-21 14:59:46.846394636  [INFO] Starting go2rtc healthcheck service...

FFprobe output from your camera

ffprobe command not found in this version

Install method

Docker Compose

Object Detector

Coral

Network connection

Wired

Camera make and model

Reolink Video Doorbell Wi-Fi, 2x Amcrest IP5M-T1277EW-AI

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

If I stop passing through the iGPU, the error messages in question stop appearing.

This is the documentation that I used to enable split iGPU passthrough and then worked on the Linux Virtual Machine iGPU passthrough Configuration section.
https://3os.org/infrastructure/proxmox/gpu-passthrough/igpu-split-passthrough/

[NickM-27]

You may want to try using preset-intel-qsv-h264 instead of vaapi