[Support]: High memory consumption, detections stop after 24 hours

[nkfs]

Checklist

• I have updated to the latest available Frigate version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.

Describe the problem you are having

I am running frigate 0.14.1-f4f3cfa using docker on a linux server. The server has 32GB of RAM (approx 20GB available to frigate), with an attached USB Coral and Nvidia GPU. It is hosting 12 1080p cameras via go2rtc restream; 6 cameras are running detect/record, the other 6 are just for live view (no record / detect). Everything works as expected when the container is started, but over the course of 24 hours, it consumes all available ram causing the OOM killer to terminate ffmpeg processes. Then errors about the coral show up ([Coral is very slow (173224939361.34 ms)])j, and detection no longer works until I restart the container. The rest of frigate seems to work as expected, my live streams still display, and connection to home assistant is still fine.

Steps to reproduce

1.Start container
2.Wait approximately 24 hours
3.Detections stop occuring
...

Version

0.14.1-f4f3cfa

In which browser(s) are you experiencing the issue with?

Chrome Version 130.0.6723.69 (Official Build) (64-bit)

Frigate config file

mqtt:
  host: 192.168.1.xxx
  user: usermqtt
  password: usermqtt
timestamp_style:
  position: bl

ffmpeg:
  output_args:
    record: preset-record-generic-audio-copy

go2rtc:
  webrtc:
    candidates: 
      - stun:8555
  streams:
   Doorbell:
    - rtsp://admin:[email protected]:8554/Streaming/Channels/101#backchannel=0
    - "ffmpeg:Doorbell#video=h264#audio=aac#width=1920#hardware"
   FrontStair:
    - rtsp://192.168.1.103:554/s0#backchannel=0
   Driveway:
    - rtsp://192.168.1.104:554/s0#backchannel=0
   FrontDoor:
    - rtsp://192.168.1.102:554/s0#backchannel=0
   LivingRoom:
    - rtsp://192.168.1.100:554/s0#backchannel=0
   PianoRoom:
    - rtsp://192.168.1.101:554/s0#backchannel=0
   garage:
    - rtsp://admin:[email protected]:554/Streaming/Channels/1#backchannel=0
   Basement:
    - rtsp://admin:[email protected]:554/Streaming/Channels/1#backchannel=0
   familyroom:
    - rtsp://admin:[email protected]:554/Streaming/Channels/1#backchannel=0
   side:
    - rtsp://admin:[email protected]/live#backchannel=0
    - "ffmpeg:side#media=video"
   camera2:
    - rtsp://admin:[email protected]/cam/realmonitor?channel=1&subtype=0#backchannel=0
    - "ffmpeg:camera2#width=1920"
   babycam:
    - rtsp://babycam:[email protected]/live#backchannel=0
    - "ffmpeg:babycam#audio=aac"
   


cameras:
  Doorbell:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Doorbell
          roles:
            - record
          input_args: preset-rtsp-restream
        - path: rtsp://admin:[email protected]:8554/Streaming/Channels/102
          roles:
            - detect
          hwaccel_args: preset-nvidia-h264
      output_args:
        record: preset-record-generic-audio-copy
    mqtt:
      timestamp: true
      bounding_box: false
      crop: true
      quality: 100
      required_zones:
        - private
    detect:
      enabled: true
      width: 640
      height: 360
    best_image_timeout: 60

    motion:
      mask:
        - 640,0,640,360,491,211,149,215,114,360,0,360,0,0
    zones:
      private:
        coordinates: 505,312,640,360,0,360,108,327

    review:
      alerts:
        required_zones: private
  FrontStair:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/FrontStair
          roles:
            - record
          input_args: preset-rtsp-restream
        - path: rtsp://192.168.1.103:554/s2
          roles:
            - detect
          hwaccel_args: preset-nvidia-h264
    detect:
      width: 1024
      height: 576
      enabled: true
    birdseye:
      mode: continuous

  Driveway:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Driveway
          roles:
            - record
          input_args: preset-rtsp-restream
        - path: rtsp://192.168.1.104:554/s2
          roles:
            - detect
          hwaccel_args: preset-nvidia-h264
    detect:
      width: 1024
      height: 576
      enabled: true

    birdseye:
      mode: continuous

    zones:
      garbage:
        coordinates: 1024,328,1024,63,776,68,728,240

  FrontDoor:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/FrontDoor
          roles:
            - record
          input_args: preset-rtsp-restream
        - path: rtsp://192.168.1.102:554/s2
          roles:
            - detect
          hwaccel_args: preset-nvidia-h264
    detect:
      width: 1024
      height: 576
      enabled: true

  LivingRoom:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/LivingRoom
          input_args: preset-rtsp-restream
    record:
      enabled: false
    detect:
      width: 1920
      height: 1080

  PianoRoom:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/PianoRoom
          input_args: preset-rtsp-restream
    record:
      enabled: false
    detect:
      width: 1920
      height: 1080

  Basement:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Basement
          input_args: preset-rtsp-restream
    record:
      enabled: false
    detect:
      width: 1920
      height: 1080

  side:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/side
          roles:
            - detect
            - record
          input_args: preset-rtsp-restream
      output_args:
        record: preset-record-generic
    detect:
      width: 1920
      height: 1080
      enabled: true
      fps: 5
    objects:
      track:
        - person


  camera2:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera2
          input_args: preset-rtsp-restream
    record:
      enabled: false

  babycam:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/babycam
          input_args: preset-rtsp-restream
    record:
      enabled: false


  familyroom:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/familyroom
          input_args: preset-rtsp-restream
    record:
      enabled: false


  garage:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://admin:[email protected]:554/Streaming/Channels/2
          hwaccel_args: preset-nvidia-h264
          roles:
            - detect
    detect:
      width: 704
      height: 480
      enabled: true
    objects:
      track:
        - person



record:
  enabled: true
  retain:
    days: 5
  events:
    retain:
      default: 14
      mode: active_objects

detect:
  enabled: false

objects:
  track:
    - person
    - car
    - motorcycle
    - bicycle
    - handbag
    - bus
    - backpack
  filters:
    person:
      min_area:.1.
      min_score: 0.6

motion:
  threshold: 40
  frame_alpha: 0.35

detectors:
  coral:
    type: edgetpu
    device: usb


birdseye:
  enabled: true
  restream: true
  mode: objects
  width: 1920
  height: 1080
version: 0.14

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    shm_size: "2gb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /mnt/frigate:/config
      - /mnt/frigate/trt-models:/trt-models
      - /mnt/frigate/storage:/media/frigate
      - type: tmpfs # Optional: 8GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 8000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "1984:1984" # go2RTC
      - "8555:8555" # WebRTC
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Relevant Frigate log output

The only interesting log snippets are these:

unknown
2024-11-22 08:58:55
unknown
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fb5a81a9900] moov atom not found
unknown
2024-11-22 08:58:55
unknown
[ERROR:[email protected]] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
unknown
2024-11-22 09:08:48
unknown
24-11-22 08:58:55.467540179
unknown
2024-11-22 08:58:55
unknown
OpenCV(4.9.0) /io/opencv/modules/videoio/src/cap_images.cpp:274: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
unknown
2024-11-22 09:08:48
unknown
24-11-22 08:58:55.467548279
unknown
2024-11-22 09:08:48
unknown
24-11-22 08:58:55.467551179

Relevant go2rtc log output

No relevant erros in go2rtc

Operating system

Other Linux

Install method

Docker Compose

Network connection

Wired

Camera make and model

Mix of hikvision, unifi g3 (non protect), and generic rtsp

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[NickM-27]

it would be good to understand specifically what process is being killed for OOM

[nkfs]

It just says "ffmpeg", is there a way to get more details than what is printed on screen?

[NickM-27]

usually there is a PID and that can be checked against ps

[MilanNeth]

I have a similar problem on HAOS friget running as add-on. Friget starts and after 24h memory spikes and i have to reboot the host. Logs are no help just that ffmpeg crashed.