[Detector Support]: ONNX Using CPU over GPU in beta (I think) #15010

distinctjuggle · 2024-11-15T20:41:41Z

distinctjuggle
Nov 15, 2024

Describe the problem you are having

This conversation is likely relevant

Picking up from above, I've got my second instance of Frigate setup and running. Initially, I thought I had GPU ONNX detections working, but now I'm questioning if that is still the case. I'm getting inference speeds of 20-30 ms instead of ~10 ms like I (thought I) was initially, and my CPU spikes to 600% reported usage in Frigate on occasion for ONNX detection. My actual total CPU CPU usage is easily >50% pretty regularly, and I believe (but have not confirmed) that it's going higher on occasion.

The startup logs don't seem to indicate one way or the other if they're using CPU or GPU for detections, nor does the detector page. I realize this is a beta, but is this intentional? I'd be happy to open a feature request issue for this if it isn't already planned.

Do I need the tensorrt image to get working Nvidia acceleration? Since I was trying ONNX directly, I didn't setup tensorrt for this instance. The custom library loading as discussed in the above discussion isn't enabled in this config, but it's the same operating system and general setup otherwise.

Is there a sure-fire way to test if ONNX is using the CPU or the GPU? Thanks.

As I wrote this, I restarted Frigate and began with 10 ms inference times again, and 0% CPU usage. Shortly after, it again jumped to 20-30 ms inference times and 600% CPU usage on the Detector CPU Usage section - so something is definitely causing it to intermittently come and go for GPU usage. For comparison, my other system has 1.6% CPU usage on the corresponding tab. I understand I'm using different generations of GPUs for this, but this seems like a heck of a difference in performance.

Version

0.15.0 Beta 1

Frigate config file

mqtt:
  enabled: true
  host: 127.0.0.1
  port: 1883
  topic_prefix: frigate
  client_id: frigate_server



go2rtc:


  api:
    tls_listen: :443
    tls_cert: |        
      -----BEGIN CERTIFICATE-----
      -----END CERTIFICATE-----
    tls_key: |         
      -----BEGIN RSA PRIVATE KEY-----
      -----END RSA PRIVATE KEY-----


  streams:
   buncha streams



  webrtc:
    candidates:
      - 192.168.10.9:8555
      - stun:8555



cameras:
  buncha cameras
ffmpeg:
  hwaccel_args: preset-nvidia-h264
  input_args: preset-rtsp-restream
  output_args:
    record: preset-record-generic-audio-copy



detectors:
  onnx:
    type: onnx


model:
  model_type: yolonas
  width: 320
  height: 320
  input_tensor: nchw
  input_pixel_format: bgr
  path: /config/model_cache/yolo_nas_s.onnx
  labelmap_path: /labelmap/coco-80.txt

detect:
 fps: 5
 

record:
  enabled: true
  expire_interval: 60
  sync_recordings: true
  retain:
    days: 15
    mode: all
  alerts:
    retain:
      days: 10
    pre_capture: 3
    post_capture: 7
  detections:
    retain:
      days: 10
    pre_capture: 3
    post_capture: 7

snapshots:
  enabled: true
  timestamp: true
  bounding_box: true
  crop: true
  retain:
    # Required: Default retention days (default: shown below)
    default: 10
    # Optional: Per object retention days
    objects:
      person: 15
  # Optional: quality of the encoded jpeg, 0-100 (default: shown below)
  quality: 100


auth:
  enabled: true
  reset_admin_password: false


version: 0.15-0

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.15.0-beta1
    shm_size: "1500mb" # update for your cameras based on calculation above
    devices:
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /srv/frigate/config/:/config
      - /srv/frigate/media:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
      - "443:443"
      - "1984:1984"
#    environment:
#      - LD_LIBRARY_PATH=/usr/local/lib/python3.9/dist-packages/tensorrt:/usr/local/cuda/lib64:/usr/local/lib/python3.9/dist-packages/nvidia/cufft/lib
    mem_swappiness: 0
    memswap_limit: 0M
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              #device_ids: ['0'] # this is only needed when using multiple GPUs
              count: 1 # number of GPUs
              capabilities: [gpu]

Relevant Frigate log output

frigate  | s6-rc: info: service frigate successfully stopped
frigate  | s6-rc: info: service go2rtc: stopping
frigate  | s6-rc: info: service frigate-log: stopping
frigate  | 2024-11-15 14:59:58.006252251  exit with signal: terminated
frigate  | s6-rc: info: service frigate-log successfully stopped
frigate  | 2024-11-15 14:59:58.024001971  [INFO] The go2rtc service exited with code 0 (by signal 0)
frigate  | s6-rc: info: service go2rtc successfully stopped
frigate  | s6-rc: info: service go2rtc-log: stopping
frigate  | s6-rc: info: service go2rtc-log successfully stopped
frigate  | s6-rc: info: service log-prepare: stopping
frigate  | s6-rc: info: service s6rc-fdholder: stopping
frigate  | s6-rc: info: service log-prepare successfully stopped
frigate  | s6-rc: info: service legacy-cont-init: stopping
frigate  | s6-rc: info: service s6rc-fdholder successfully stopped
frigate  | s6-rc: info: service legacy-cont-init successfully stopped
frigate  | s6-rc: info: service fix-attrs: stopping
frigate  | s6-rc: info: service fix-attrs successfully stopped
frigate  | s6-rc: info: service s6rc-oneshot-runner: stopping
frigate  | s6-rc: info: service s6rc-oneshot-runner successfully stopped
frigate  | s6-rc: info: service s6rc-fdholder: starting
frigate  | s6-rc: info: service s6rc-oneshot-runner: starting
frigate  | s6-rc: info: service s6rc-oneshot-runner successfully started
frigate  | s6-rc: info: service fix-attrs: starting
frigate  | s6-rc: info: service s6rc-fdholder successfully started
frigate  | s6-rc: info: service fix-attrs successfully started
frigate  | s6-rc: info: service legacy-cont-init: starting
frigate  | s6-rc: info: service legacy-cont-init successfully started
frigate  | s6-rc: info: service log-prepare: starting
frigate  | s6-rc: info: service log-prepare successfully started
frigate  | s6-rc: info: service nginx-log: starting
frigate  | s6-rc: info: service go2rtc-log: starting
frigate  | s6-rc: info: service frigate-log: starting
frigate  | s6-rc: info: service certsync-log: starting
frigate  | s6-rc: info: service frigate-log successfully started
frigate  | s6-rc: info: service nginx-log successfully started
frigate  | s6-rc: info: service go2rtc-log successfully started
frigate  | s6-rc: info: service go2rtc: starting
frigate  | s6-rc: info: service certsync-log successfully started
frigate  | s6-rc: info: service go2rtc successfully started
frigate  | s6-rc: info: service go2rtc-healthcheck: starting
frigate  | s6-rc: info: service frigate: starting
frigate  | s6-rc: info: service frigate successfully started
frigate  | s6-rc: info: service nginx: starting
frigate  | s6-rc: info: service go2rtc-healthcheck successfully started
frigate  | 2024-11-15 15:00:02.967366359  [INFO] Preparing new go2rtc config...
frigate  | 2024-11-15 15:00:02.967649909  [INFO] Preparing Frigate...
frigate  | 2024-11-15 15:00:02.968084081  [INFO] Starting NGINX...
frigate  | 2024-11-15 15:00:02.970763281  [INFO] Starting Frigate...
frigate  | s6-rc: info: service nginx successfully started
frigate  | s6-rc: info: service certsync: starting
frigate  | s6-rc: info: service certsync successfully started
frigate  | s6-rc: info: service legacy-services: starting
frigate  | 2024-11-15 15:00:03.151138383  [INFO] Starting certsync...
frigate  | s6-rc: info: service legacy-services successfully started
frigate  | 2024-11-15 15:00:03.286676256  127.0.0.1 - - [15/Nov/2024:15:00:03 -0500] "" 400 0 "-" "-" "-"
frigate  | 2024-11-15 15:00:03.306667096  [INFO] Not injecting WebRTC candidates into go2rtc config as it has been set manually
frigate  | 2024-11-15 15:00:03.331734007  [INFO] Starting go2rtc...
frigate  | 2024-11-15 15:00:03.414393515  15:00:03.414 INF go2rtc platform=linux/amd64 revision=b2399f3 version=1.9.2
frigate  | 2024-11-15 15:00:03.414396424  15:00:03.414 INF config path=/dev/shm/go2rtc.yaml
frigate  | 2024-11-15 15:00:03.414732677  15:00:03.414 INF [api] listen addr=:1984
frigate  | 2024-11-15 15:00:03.414966053  15:00:03.414 INF [api] tls listen addr=:443
frigate  | 2024-11-15 15:00:03.415277140  15:00:03.415 INF [rtsp] listen addr=:8554
frigate  | 2024-11-15 15:00:03.415595233  15:00:03.415 INF [webrtc] listen addr=:8555
frigate  | 2024-11-15 15:00:03.891895888  2024/11/15 15:00:03 http: TLS handshake error from 127.0.0.1:54462: local error: tls: bad record MAC
frigate  | 2024-11-15 15:00:04.794608614  [2024-11-15 15:00:04] frigate.util.config            INFO    : Checking if frigate config needs migration...
frigate  | 2024-11-15 15:00:04.833956096  [2024-11-15 15:00:04] frigate.util.config            INFO    : frigate config does not need migration...
frigate  | 2024-11-15 15:00:07.944093200  2024/11/15 15:00:07 [error] 162#162: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:07.944108633  2024/11/15 15:00:07 [error] 162#162: *2 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:08.629665528  2024/11/15 15:00:08 [error] 163#163: *4 connect() failed (111: Connection refused) while connecting to upstream, client: client_endpoint, server: , request: "GET /ws HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "ip_endpoint"
frigate  | 2024-11-15 15:00:08.629693788  2024/11/15 15:00:08 [error] 163#163: *4 auth request unexpected status: 502 while sending to client, client: client_endpoint, server: , request: "GET /ws HTTP/1.1", host: "ip_endpoint"
frigate  | 2024-11-15 15:00:08.629794540  client_endpoint - - [15/Nov/2024:15:00:08 -0500] "GET /ws HTTP/1.1" 500 177 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0" "-"
frigate  | 2024-11-15 15:00:12.968171653  [INFO] Starting go2rtc healthcheck service...
frigate  | 2024-11-15 15:00:12.977093251  2024/11/15 15:00:12 [error] 163#163: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:12.977096592  2024/11/15 15:00:12 [error] 163#163: *6 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:13.954699624  2024/11/15 15:00:13 [error] 164#164: *8 connect() failed (111: Connection refused) while connecting to upstream, client: client_endpoint, server: , request: "GET /api/profile HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "ip_endpoint", referrer: "https://ip_endpoint/config"
frigate  | 2024-11-15 15:00:13.954726259  2024/11/15 15:00:13 [error] 164#164: *8 auth request unexpected status: 502 while sending to client, client: client_endpoint, server: , request: "GET /api/profile HTTP/1.1", host: "ip_endpoint", referrer: "https://ip_endpoint/config"
frigate  | 2024-11-15 15:00:13.954823149  client_endpoint - - [15/Nov/2024:15:00:13 -0500] "GET /api/profile HTTP/1.1" 500 177 "https://ip_endpoint/config" "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0" "-"
frigate  | 2024-11-15 15:00:14.056944353  2024/11/15 15:00:14 [error] 165#165: *9 connect() failed (111: Connection refused) while connecting to upstream, client: client_endpoint, server: , request: "GET /ws HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "ip_endpoint"
frigate  | 2024-11-15 15:00:14.056947322  2024/11/15 15:00:14 [error] 165#165: *9 auth request unexpected status: 502 while sending to client, client: client_endpoint, server: , request: "GET /ws HTTP/1.1", host: "ip_endpoint"
frigate  | 2024-11-15 15:00:14.057001462  client_endpoint - - [15/Nov/2024:15:00:14 -0500] "GET /ws HTTP/1.1" 500 177 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0" "-"
frigate  | 2024-11-15 15:00:18.008116399  2024/11/15 15:00:18 [error] 164#164: *12 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:18.008119406  2024/11/15 15:00:18 [error] 164#164: *12 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:18.459710804  2024/11/15 15:00:18 [error] 162#162: *14 connect() failed (111: Connection refused) while connecting to upstream, client: client_endpoint, server: , request: "GET /api/config/raw HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "ip_endpoint", referrer: "https://ip_endpoint/config"
frigate  | 2024-11-15 15:00:18.459715591  2024/11/15 15:00:18 [error] 162#162: *14 auth request unexpected status: 502 while sending to client, client: client_endpoint, server: , request: "GET /api/config/raw HTTP/1.1", host: "ip_endpoint", referrer: "https://ip_endpoint/config"
frigate  | 2024-11-15 15:00:18.459763434  client_endpoint - - [15/Nov/2024:15:00:18 -0500] "GET /api/config/raw HTTP/1.1" 500 177 "https://ip_endpoint/config" "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0" "-"
frigate  | 2024-11-15 15:00:19.893392140  2024/11/15 15:00:19 [error] 162#162: *16 connect() failed (111: Connection refused) while connecting to upstream, client: client_endpoint, server: , request: "GET /ws HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "ip_endpoint"
frigate  | 2024-11-15 15:00:19.893395530  2024/11/15 15:00:19 [error] 162#162: *16 auth request unexpected status: 502 while sending to client, client: client_endpoint, server: , request: "GET /ws HTTP/1.1", host: "ip_endpoint"
frigate  | 2024-11-15 15:00:19.893517939  client_endpoint - - [15/Nov/2024:15:00:19 -0500] "GET /ws HTTP/1.1" 500 177 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0" "-"
frigate  | 2024-11-15 15:00:23.043058146  2024/11/15 15:00:23 [error] 165#165: *18 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:23.043088146  2024/11/15 15:00:23 [error] 165#165: *18 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:23.579826269  2024/11/15 15:00:23 [error] 162#162: *20 connect() failed (111: Connection refused) while connecting to upstream, client: client_endpoint, server: , request: "GET /api/config HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "ip_endpoint", referrer: "https://ip_endpoint/config"
frigate  | 2024-11-15 15:00:23.579829623  2024/11/15 15:00:23 [error] 162#162: *20 auth request unexpected status: 502 while sending to client, client: client_endpoint, server: , request: "GET /api/config HTTP/1.1", host: "ip_endpoint", referrer: "https://ip_endpoint/config"
frigate  | 2024-11-15 15:00:23.579913192  client_endpoint - - [15/Nov/2024:15:00:23 -0500] "GET /api/config HTTP/1.1" 500 177 "https://ip_endpoint/config" "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0" "-"
frigate  | 2024-11-15 15:00:25.627804540  2024/11/15 15:00:25 [error] 162#162: *22 connect() failed (111: Connection refused) while connecting to upstream, client: client_endpoint, server: , request: "GET /ws HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "ip_endpoint"
frigate  | 2024-11-15 15:00:25.627822607  2024/11/15 15:00:25 [error] 162#162: *22 auth request unexpected status: 502 while sending to client, client: client_endpoint, server: , request: "GET /ws HTTP/1.1", host: "ip_endpoint"
frigate  | 2024-11-15 15:00:25.628006307  client_endpoint - - [15/Nov/2024:15:00:25 -0500] "GET /ws HTTP/1.1" 500 177 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0" "-"
frigate  | 2024-11-15 15:00:28.086183660  2024/11/15 15:00:28 [error] 162#162: *24 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:28.086186770  2024/11/15 15:00:28 [error] 162#162: *24 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:30.726645740  [2024-11-15 15:00:30] frigate.app                    INFO    : Starting Frigate (0.15.0-33825f6)
frigate  | 2024-11-15 15:00:30.738750667  [2024-11-15 15:00:30] peewee_migrate.logs            INFO    : Starting migrations
frigate  | 2024-11-15 15:00:30.739346976  [2024-11-15 15:00:30] peewee_migrate.logs            INFO    : There is nothing to migrate
frigate  | 2024-11-15 15:00:30.749144594  [2024-11-15 15:00:30] frigate.app                    INFO    : Recording process started: 478
frigate  | 2024-11-15 15:00:30.749250221  [2024-11-15 15:00:30] frigate.app                    INFO    : Review process started: 485
frigate  | 2024-11-15 15:00:30.751280646  [2024-11-15 15:00:30] frigate.app                    INFO    : go2rtc process pid: 98
frigate  | 2024-11-15 15:00:30.766063968  [2024-11-15 15:00:30] detector.onnx                  INFO    : Starting detection process: 495
frigate  | 2024-11-15 15:00:30.766154923  [2024-11-15 15:00:30] frigate.detectors.plugins.onnx INFO    : ONNX: loaded onnxruntime module
frigate  | 2024-11-15 15:00:30.766206314  [2024-11-15 15:00:30] frigate.detectors.plugins.onnx INFO    : ONNX: loading /config/model_cache/yolo_nas_s.onnx
frigate  | 2024-11-15 15:00:30.777325321  [2024-11-15 15:00:30] frigate.app                    INFO    : Output process started: 516
frigate  | 2024-11-15 15:00:30.809147782  [2024-11-15 15:00:30] frigate.app                    INFO    : Camera processor started for cam1: 535
frigate  | 2024-11-15 15:00:30.816891318  [2024-11-15 15:00:30] frigate.app                    INFO    : Camera processor started for cam2: 536
frigate  | 2024-11-15 15:00:30.831827355  [2024-11-15 15:00:30] frigate.app                    INFO    : Camera processor started for cam3: 539
frigate  | 2024-11-15 15:00:30.832734223  [2024-11-15 15:00:30] frigate.app                    INFO    : Camera processor started for cam4: 552
frigate  | 2024-11-15 15:00:30.840185820  [2024-11-15 15:00:30] frigate.app                    INFO    : Camera processor started for cam5: 553
frigate  | 2024-11-15 15:00:30.853423006  [2024-11-15 15:00:30] frigate.app                    INFO    : Camera processor started for cam6: 568
frigate  | 2024-11-15 15:00:30.853598659  [2024-11-15 15:00:30] frigate.app                    INFO    : Capture process started for cam1: 569
frigate  | 2024-11-15 15:00:30.859558694  [2024-11-15 15:00:30] frigate.app                    INFO    : Capture process started for cam2: 586
frigate  | 2024-11-15 15:00:30.865240902  [2024-11-15 15:00:30] frigate.app                    INFO    : Capture process started for cam3: 590
frigate  | 2024-11-15 15:00:30.871576196  [2024-11-15 15:00:30] frigate.app                    INFO    : Capture process started for cam4: 595
frigate  | 2024-11-15 15:00:30.886023395  [2024-11-15 15:00:30] frigate.app                    INFO    : Capture process started for cam5: 601
frigate  | 2024-11-15 15:00:30.891803103  [2024-11-15 15:00:30] frigate.app                    INFO    : Capture process started for cam6: 607
frigate  | 2024-11-15 15:00:30.930140814  [2024-11-15 15:00:30] frigate.detectors.plugins.onnx INFO    : ONNX: /config/model_cache/yolo_nas_s.onnx loaded
frigate  | 2024-11-15 15:00:31.975598635  2024/11/15 15:00:31 [error] 162#162: *26 connect() failed (111: Connection refused) while connecting to upstream, client: client_endpoint, server: , request: "GET /ws HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "ip_endpoint"
frigate  | 2024-11-15 15:00:31.975603730  2024/11/15 15:00:31 [error] 162#162: *26 auth request unexpected status: 502 while sending to client, client: client_endpoint, server: , request: "GET /ws HTTP/1.1", host: "ip_endpoint"
frigate  | 2024-11-15 15:00:31.975621482  client_endpoint - - [15/Nov/2024:15:00:31 -0500] "GET /ws HTTP/1.1" 500 177 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0" "-"
frigate  | 2024-11-15 15:00:33.121013282  2024/11/15 15:00:33 [error] 162#162: *28 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:33.121026983  2024/11/15 15:00:33 [error] 162#162: *28 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", host: "127.0.0.1:5000"
frigate  | 2024-11-15 15:00:34.908220846  [2024-11-15 15:00:34] frigate.api.fastapi_app        INFO    : Starting FastAPI app
frigate  | 2024-11-15 15:00:35.061612619  [2024-11-15 15:00:35] frigate.api.fastapi_app        INFO    : FastAPI started
frigate  | 2024-11-15 15:00:35.357884360  [2024-11-15 15:00:35] frigate.record.util            INFO    : Deleting 3161 recording DB entries with missing files
frigate  | 2024-11-15 15:00:39.563235074  client_endpoint - - [15/Nov/2024:15:00:39 -0500] "GET /api/profile HTTP/1.1" 200 20 "https://ip_endpoint/config" "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0" "-"
frigate  | 2024-11-15 15:00:54.921757531  client_endpoint - - [15/Nov/2024:15:00:54 -0500] "GET /api/config HTTP/1.1" 200 35848 "https://ip_endpoint/config" "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0" "-"

Install method

Docker Compose

Object Detector

Other

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

"Hardware info":

{"0":{"name":"NVIDIAGeForceGTX1070","driver":"565.57.01","cuda_compute":6.1,"vbios":"86.04.50.40.BD"}}

Answered by distinctjuggle

Nov 15, 2024

On the host I get:

# nvidia-smi
Fri Nov 15 15:52:33 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+=============…

View full answer

NickM-27 · 2024-11-15T20:44:45Z

NickM-27
Nov 15, 2024
Collaborator Sponsor

the system metrics you are showing only shows the startup, which uses CPU to convert the onnx model into the correct format to be loaded onto the GPU, so you can't go off of the inference speed or the CPU usage during this initial startup

5 replies

NickM-27 Nov 15, 2024
Collaborator Sponsor

it uses the CPU on startup to convert the model, once the model is fully loaded on the GPU then the CPU usage reduces, which in my setup takes ~20 seconds or so

NickM-27 Nov 15, 2024
Collaborator Sponsor

Do I need the tensorrt image to get working Nvidia acceleration? Since I was trying ONNX directly, I didn't setup tensorrt for this instance. The custom library loading as discussed in the above discussion isn't enabled in this config, but it's the same operating system and general setup otherwise.

yes you do need the tensorrt variant of the builds

Is there a sure-fire way to test if ONNX is using the CPU or the GPU? Thanks.

check nvidia-smi on the host and see if frigate.detector.onnx is in the list

distinctjuggle Nov 15, 2024
Author

Mine is never reducing. The screenshots show a 12+ hour uptime I believe.

At startup is where I see the 10 ms inference speeds and nearly 0% CPU usage for this machine.

Meanwhile my other system is at 10-11 ms inference speed at about 2% GPU usage and easily <10% ONNX usage.

distinctjuggle Nov 15, 2024
Author

On the host I get:

# nvidia-smi
Fri Nov 15 15:52:33 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1070        Off |   00000000:01:00.0  On |                  N/A |
| 34%   51C    P2             41W /  151W |    1434MiB /   8192MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A    693106      C   /usr/lib/ffmpeg/7.0/bin/ffmpeg                240MiB |
|    0   N/A  N/A    693117      C   /usr/lib/ffmpeg/7.0/bin/ffmpeg                186MiB |
|    0   N/A  N/A    693132      C   /usr/lib/ffmpeg/7.0/bin/ffmpeg                322MiB |
|    0   N/A  N/A    693135      C   /usr/lib/ffmpeg/7.0/bin/ffmpeg                174MiB |
|    0   N/A  N/A    693157      C   /usr/lib/ffmpeg/7.0/bin/ffmpeg                312MiB |
|    0   N/A  N/A    693159      C   /usr/lib/ffmpeg/7.0/bin/ffmpeg                186MiB |
+-----------------------------------------------------------------------------------------+

To me, this just looks like the camera's hardware acceleration and not ONNX. I'll try loading the tensorrt image and go from there, but that'll take a long time to download now that I've already deployed the server.

Edit: The tensorrt image seems to have fixed things. I'll make an issue so that this gets added to the documentation, as it currently isn't present. I also think a built-in screen somewhere within Frigate to tell the user what device the detection method/ONNX is using would be helpful.

Answer selected by NickM-27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Detector Support]: ONNX Using CPU over GPU in beta (I think) #15010

{{title}}

Replies: 1 comment 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[Detector Support]: ONNX Using CPU over GPU in beta (I think) #15010

distinctjuggle Nov 15, 2024

Describe the problem you are having

Version

Frigate config file

docker-compose file or Docker CLI command

Relevant Frigate log output

Install method

Object Detector

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Replies: 1 comment · 5 replies

NickM-27 Nov 15, 2024 Collaborator Sponsor

NickM-27 Nov 15, 2024 Collaborator Sponsor

NickM-27 Nov 15, 2024 Collaborator Sponsor

distinctjuggle Nov 15, 2024 Author

distinctjuggle Nov 15, 2024 Author

distinctjuggle
Nov 15, 2024

Replies: 1 comment 5 replies

NickM-27
Nov 15, 2024
Collaborator Sponsor

NickM-27 Nov 15, 2024
Collaborator Sponsor

NickM-27 Nov 15, 2024
Collaborator Sponsor

distinctjuggle Nov 15, 2024
Author

distinctjuggle Nov 15, 2024
Author