[HW Accel Support]: Crash the hole Proxmox server with Hardware Acceleration #10782

Lip90 · 2024-04-01T20:40:59Z

Lip90
Apr 1, 2024

Describe the problem you are having

I use Frigate on a Proxmox LXC under docker. As soon as I enable hardware acceleration. Crashed the whole system and proxmox and all vm and lxc are restarted. The problem only occurs with frigate. In Jellyfin on the same lcx container the hardware acceleration works without problems. This is generally the first time an LXC container has crashed my entire server.

Does anyone have an idea what the problem could be. I have googled a lot but found nothing.

Version

0.13.2-6476F8A

Frigate config file

mqtt:
  enabled: False

#ffmpeg:
#  hwaccel_args: preset-vaapi

cameras:
  Eingang:
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: rtsp://127.0.0.1:8554/Eingang
          input_args: preset-rtsp-restream
          roles:
            - record

go2rtc:
  streams:
    #Eingang: onvif://homeassistant:[email protected]:8000
    Eingang: rtsp://homeassistant:[email protected]:554/h264Preview_01_main

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    #restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "128mb" # update for your cameras based on calculation above
    devices:
#      - /dev/bus/usb:/dev/bus/usb  # Passes the USB Coral, needs to be modified for other versions
#      - /dev/apex_0:/dev/apex_0    # Passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
#      - /dev/video11:/dev/video11  # For Raspberry Pi 4B
      - /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your hardware
#      - /dev/dri/card1:/dev/dri/card1
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/frigate:/config
      - /nfs/frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "hanspetersepp"
      LIBVA_DRIVER_NAME: radeonsi

Relevant log output

Can't see a log because of the direkt crash.

FFprobe output from your camera

# ffprobe rtsp://homeassistant:[email protected]:554/h264Preview_01_main
ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, rtsp, from 'rtsp://homeassistant:[email protected]:554/h264Preview_01_main':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h264Preview_01_main
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 2560x1440, 12.50 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

Operating system

Proxmox

Install method

Docker Compose

Network connection

Mixed

Camera make and model

Reolink RLC-410W

Any other information that may be helpful

No response

NickM-27 · 2024-04-01T20:47:09Z

NickM-27
Apr 1, 2024
Collaborator Sponsor

what is the output of vainfo run from the lxc?

2 replies

Lip90 Apr 10, 2024
Author

@NickM-27 Have you got an idea?

NickM-27 Apr 10, 2024
Collaborator Sponsor

Nope, maybe something proxmox related which I have no experience with. I run frigate with the same driver and an AMD 5700G and have not had any issues

Lip90 · 2024-04-01T20:49:06Z

Lip90
Apr 1, 2024
Author

root@debian-docker:~# vainfo
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
error: can't connect to X server!
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.12.0)
vainfo: Driver version: Mesa Gallium driver 22.3.6 for AMD Radeon Graphics (renoir, LLVM 15.0.6, DRM 3.54, 6.5.13-3-pve)
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc

0 replies

Lip90 · 2024-04-10T15:23:20Z

Lip90
Apr 10, 2024
Author

On a proxmox host or bearmetal?

…

________________________________ Von: Nicolas Mowen ***@***.***> Gesendet: Wednesday, April 10, 2024 5:20:11 PM An: blakeblackshear/frigate ***@***.***> Cc: Philipp Stiastny-Pöhlmann ***@***.***>; Author ***@***.***> Betreff: Re: [blakeblackshear/frigate] [HW Accel Support]: Crash the hole Proxmox server with Hardware Acceleration (Discussion #10782) Nope, maybe something proxmox related which I have no experience with. I run frigate with the same driver and an AMD 5700G and have not had any issues — Reply to this email directly, view it on GitHub<#10782 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQKB62DEHHA3M3FBOPB5FFLY4VKCXAVCNFSM6AAAAABFSFRRUCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TANZSHA3DQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

4 replies

NickM-27 Apr 10, 2024
Collaborator Sponsor

baremetal, like I said I have no experience with proxmox

ch1988 Apr 10, 2024

Same with my proxmox. Once a week, the whole systems "hangs". No error, nothing - just a freeze.

kafisc1 Apr 19, 2024

Same here.

thomas-oprews Apr 24, 2024

same for me, after 10 min up

tdashmike · 2024-06-30T15:51:33Z

tdashmike
Jun 30, 2024

I haven’t had time to try this but maybe it’s the kernel although the article is talking about vGPU/DKMS kernel issues. My host crashes as well running Frigate in a container on a Ubuntu 24.04 VM.

https://www.derekseaman.com/2023/11/proxmox-ve-8-1-windows-11-vgpu-vt-d-passthrough-with-intel-alder-lake.html

0 replies

Lip90 · 2024-09-23T13:36:05Z

Lip90
Sep 23, 2024
Author

Has the problem gone away for you or is it still the same case?

0 replies

nickk289 · 2024-10-22T13:16:18Z

nickk289
Oct 22, 2024

Same problem for me. I have Frigate as docker container on Ubuntu 24.04 in Proxmox LXC with iGPU passtrough. Everything works as it should, but after some time (the last crash was about 5 hours after starting the LXC) the whole system freezes without any error in log and it remains stuck until I perform a hard reset with the power button.

OS: Proxmox 8.7.2
Kernel version: Linux 6.8.12-2-pve (2024-09-05T10:03Z)
CPU: Intel N100
iGPU: Intel UHD Graphics 730
Frigate: 0.14.1-f4f3cfa

Proxmox log during the crash (it crashed at about 03:30):

Oct 22 00:00:20 pve pveproxy[1335]: worker 743773 finished
Oct 22 00:00:20 pve pveproxy[1335]: worker 744276 finished
Oct 22 00:00:20 pve pveproxy[1335]: worker 740287 finished
Oct 22 00:17:01 pve CRON[796040]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 22 00:17:01 pve CRON[796041]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Oct 22 00:17:01 pve CRON[796040]: pam_unix(cron:session): session closed for user root
Oct 22 01:17:01 pve CRON[819494]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 22 01:17:01 pve CRON[819495]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Oct 22 01:17:01 pve CRON[819494]: pam_unix(cron:session): session closed for user root
Oct 22 02:17:01 pve CRON[842938]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 22 02:17:01 pve CRON[842939]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Oct 22 02:17:01 pve CRON[842938]: pam_unix(cron:session): session closed for user root
-- Reboot --
Oct 22 14:19:21 pve kernel: Linux version 6.8.12-2-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-2 (2024-09-05T10:03Z) ()
Oct 22 14:19:21 pve kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-2-pve root=/dev/mapper/pve-root ro quiet
Oct 22 14:19:21 pve kernel: KERNEL supported cpus:
Oct 22 14:19:21 pve kernel:   Intel GenuineIntel
Oct 22 14:19:21 pve kernel:   AMD AuthenticAMD
Oct 22 14:19:21 pve kernel:   Hygon HygonGenuine
Oct 22 14:19:21 pve kernel:   Centaur CentaurHauls
Oct 22 14:19:21 pve kernel:   zhaoxin   Shanghai  
Oct 22 14:19:21 pve kernel: x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks
Oct 22 14:19:21 pve kernel: BIOS-provided physical RAM map:
Oct 22 14:19:21 pve kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
Oct 22 14:19:21 pve kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
Oct 22 14:19:21 pve kernel: BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable

Frigate vainfo:

vainfo: VA-API version: 1.17 (libva 2.10.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.1.1 ()
vainfo: Supported profile and entrypoints
VAProfileNone : VAEntrypointVideoProc
VAProfileNone : VAEntrypointStats
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Simple : VAEntrypointEncSlice
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointFEI
VAProfileH264Main : VAEntrypointEncSliceLP
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointFEI
VAProfileH264High : VAEntrypointEncSliceLP
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointEncPicture
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264ConstrainedBaseline: VAEntrypointFEI
VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
VAProfileVP8Version0_3 : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointFEI
VAProfileHEVCMain : VAEntrypointEncSliceLP
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointEncSliceLP
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointEncSliceLP
VAProfileVP9Profile1 : VAEntrypointVLD
VAProfileVP9Profile1 : VAEntrypointEncSliceLP
VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointEncSliceLP
VAProfileVP9Profile3 : VAEntrypointVLD
VAProfileVP9Profile3 : VAEntrypointEncSliceLP
VAProfileHEVCMain12 : VAEntrypointVLD
VAProfileHEVCMain12 : VAEntrypointEncSlice
VAProfileHEVCMain422_10 : VAEntrypointVLD
VAProfileHEVCMain422_10 : VAEntrypointEncSlice
VAProfileHEVCMain422_12 : VAEntrypointVLD
VAProfileHEVCMain422_12 : VAEntrypointEncSlice
VAProfileHEVCMain444 : VAEntrypointVLD
VAProfileHEVCMain444 : VAEntrypointEncSliceLP
VAProfileHEVCMain444_10 : VAEntrypointVLD
VAProfileHEVCMain444_10 : VAEntrypointEncSliceLP
VAProfileHEVCMain444_12 : VAEntrypointVLD
VAProfileHEVCSccMain : VAEntrypointVLD
VAProfileHEVCSccMain : VAEntrypointEncSliceLP
VAProfileHEVCSccMain10 : VAEntrypointVLD
VAProfileHEVCSccMain10 : VAEntrypointEncSliceLP
VAProfileHEVCSccMain444 : VAEntrypointVLD
VAProfileHEVCSccMain444 : VAEntrypointEncSliceLP
VAProfileAV1Profile0 : VAEntrypointVLD
VAProfileHEVCSccMain444_10 : VAEntrypointVLD
VAProfileHEVCSccMain444_10 : VAEntrypointEncSliceLP

2 replies

SaohTG Oct 26, 2024

I think it's not stable on this support, I switched to a Dell server and it works perfectly now

iamhermes Nov 26, 2024

@nickk289
It was exactly the same for me. The system just froze and then that was it. I got myself an almost identical system again and after reinstalling it, the only hardware that was the same was a Coral TPU, a USB HDD and a DisplayPort dummy plug.

The computer lasted two weeks without stopping (thought the previous hardware had the problem). It froze again.

I had read a lot about this error online and stumbled across a post in which a computer with Ubuntu also froze. According to the post, the freeze went away immediately when the monitor (or dummy plug) was unplugged. As a precaution, I have already ordered new dummy plugs. And the strange kernel messages during startup (i915 [drm] Cannot find any crtc or sizes) also seem to point to a problem with the connected monitor/dummy.

So far I haven't had another freeze to test this out. The previously plugged in dummy plug (the most likely culprit) is now disposed of.

deanfourie1 · 2024-10-26T05:54:27Z

deanfourie1
Oct 26, 2024

Same for me,

Using Coral via USB passthrough, Also passed through the entire GPU.

Crashed every 5 mins or so.

0 replies

HDHristov · 2024-11-26T20:54:06Z

HDHristov
Nov 26, 2024

Had the same issue. Solution for the time being:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_idle.max_cstate=1 processor.max_cstate=1"
Also disable C states in bios.
I don't know why and how, but it is stable for now.
Not ideal, has the expected side effects, but the system has been stable so far.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HW Accel Support]: Crash the hole Proxmox server with Hardware Acceleration #10782

{{title}}

Replies: 8 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[HW Accel Support]: Crash the hole Proxmox server with Hardware Acceleration #10782

Describe the problem you are having

Version

Frigate config file

docker-compose file or Docker CLI command

Relevant log output

FFprobe output from your camera

Operating system

Install method

Network connection

Camera make and model

Any other information that may be helpful

Replies: 8 comments · 8 replies

NickM-27 Apr 1, 2024 Collaborator Sponsor

Lip90 Apr 10, 2024 Author

NickM-27 Apr 10, 2024 Collaborator Sponsor

Lip90 Apr 1, 2024 Author

Lip90 Apr 10, 2024 Author

NickM-27 Apr 10, 2024 Collaborator Sponsor

Lip90 Sep 23, 2024 Author

Replies: 8 comments 8 replies

NickM-27
Apr 1, 2024
Collaborator Sponsor

Lip90 Apr 10, 2024
Author

NickM-27 Apr 10, 2024
Collaborator Sponsor

Lip90
Apr 1, 2024
Author

Lip90
Apr 10, 2024
Author

NickM-27 Apr 10, 2024
Collaborator Sponsor

Lip90
Sep 23, 2024
Author