[Support]: Frigate v0.13 consuming a lot more memory than v0.12 #10005

heisenberg2980 · 2024-02-23T22:22:07Z

heisenberg2980
Feb 23, 2024

Describe the problem you are having

After updating Frigate to v0.13 it is using double amount of memory than v0.12, increasing from less than 5Gb to almost 10Gb using the same config file, and crashing when trying to see the timeline from Home Assistant.

This is the top command executed inside the container using v0.12.1, where process python3 is using 2.4% of memory:

And the same top command after the update to v0.13.2 shows process python3 using 9% of memory:

The issue might be related to the size of the database (3Gb at the moment) as it does not happen when starting Frigate v0.13 with an empty database as shown here (python3 using 1% of memory):

Version

0.13.2

Frigate config file

mqtt:
  host: 192.168.0.x
  user: xxxxxxxxx
  password: xxxxxxxxxx

detectors:
  coral:
    type: edgetpu
    device: usb

database:
  path: /db/frigate.db

objects:
  track:
    - person
    - cat
  filters:
    person:
      max_area: 125000
      threshold: 0.8
    cat:
      max_area: 30000
      threshold: 0.75

ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    record: preset-record-generic-audio-aac

birdseye:
  mode: continuous 
  width: 1920
  height: 1080


record:
  enabled: True
  retain:
    days: 30
  events:
    retain:
      default: 120

snapshots:
  retain:
    default: 10000

detect:
  enabled: True
  width: 1280 # <---- update for your camera's resolution
  height: 720 # <---- update for your camera's resolution
  fps: 5
  stationary:
    interval: 150

motion:
  contour_area: 20
  mask:
    - 1236,29,1236,60,880,61,880,27

go2rtc:
  streams:
    cam_1:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    cam_1_sub:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=2
    cam_2:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    cam_2_sub:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=2
    cam_3:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    cam_3_sub:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=2
    cam_4:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    cam_4_sub:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=2
    cam_5:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    cam_5_sub:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=2
    cam_6:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    cam_6_sub:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=2
    cam_7:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    cam_7_sub:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=2
    cam_8:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    cam_8_sub:
      - rtsp://xxx:[email protected]:554/cam/realmonitor?channel=1&subtype=2
    cam_9:
      - rtsp://xxx:[email protected]:554/live/main
    cam_9_sub:
      - rtsp://xxx:[email protected]:554/live/sub

cameras:
  cam_1: # Main Street 
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cam_1 # <--- the name here must match the name of the camera in restream
          roles:
            - record
            - detect
    detect:
      enabled: True
      width: 2688 # <---- update for your camera's resolution
      height: 1520 # <---- update for your camera's resolution
      fps: 5
      stationary:
        interval: 150
    record: # <----- Enable recording
      enabled: True
    snapshots: # <----- Enable snapshots
      enabled: True
    mqtt: # only apply to snapshots
      enabled: False
    objects:
      track:
        - person
        - cat
        - car
      filters:
        person:
          threshold: 0.74
          mask:
            - 1449,1512,2688,1512,1848,888,1676,836
        car:
          min_area: 10000
          mask:
            - 0,0,611,0,607,160,0,334
    motion:
      mask:
        - 2600,60,2600,125,2000,125,2000,60
        - 930,1512,1905,1512,2075,1182,1590,983
        - 2014,704,2073,704,2153,336,2077,321
    zones:
      outside_main_door:
        coordinates: 714,538,0,691,0,1512,1168,1512,2161,1002,2237,785

  cam_2:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cam_2 # <--- the name here must match the name of the camera in restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/cam_2_sub # <--- the name here must match the name of the camera_sub in restream
          roles:
            - detect
    record: # <----- Enable recording
      enabled: True
    snapshots: # <----- Enable snapshots
      enabled: True
    mqtt: # only apply to snapshots
      enabled: False
    motion:
      mask:
        - 1236,29,1236,60,880,61,880,27
        - 202,720,869,720,564,436,162,568
    objects:
      filters:
        person:
          min_score: 0.8

  cam_3:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cam_3 # <--- the name here must match the name of the camera in restream
          roles:
            - record
            - detect
    detect:
      enabled: True
      width: 2688 # <---- update for your camera's resolution
      height: 1520 # <---- update for your camera's resolution
      fps: 5
      stationary:
        interval: 150
    record: # <----- Enable recording
      enabled: True
    snapshots: # <----- Enable snapshots
      enabled: True
    mqtt: # only apply to snapshots
      enabled: False
    motion:
      mask:
        - 2600,60,2600,125,2000,125,2000,60
        - 2688,0,2688,281,2538,301,2103,160,2091,0
        - 2205,1520,1574,1143,951,1520
    objects:
      mask:
        - 142,241,663,109,1379,57,1392,0,91,0
      filters:
        person:
          max_area: 600000
        cat:
          max_area: 80000

  cam_4:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cam_4 # <--- the name here must match the name of the camera in restream
          roles:
            - record
            - detect
    detect:
      enabled: True
      width: 2688 # <---- update for your camera's resolution
      height: 1520 # <---- update for your camera's resolution
      fps: 5
      stationary:
        interval: 150
    record: # <----- Enable recording
      enabled: True
    snapshots: # <----- Enable snapshots
      enabled: True
    mqtt: # only apply to snapshots
      enabled: False
    motion:
      mask:
        - 2600,60,2600,125,2000,125,2000,60
    objects:
      filters:
        person:
          threshold: 0.8

  cam_5:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cam_5 # <--- the name here must match the name of the camera in restream
          roles:
            - record
            - detect
    detect:
      enabled: True
      width: 2688 # <---- update for your camera's resolution
      height: 1520 # <---- update for your camera's resolution
      fps: 5
      stationary:
        interval: 150
    record: # <----- Enable recording
      enabled: True
    snapshots: # <----- Enable snapshots
      enabled: True
    mqtt: # only apply to snapshots
      enabled: False
    motion:
      mask:
        - 2688,0,2688,932,1032,376,997,0
    objects:
      track:
        - person
        - cat
        - car
      filters:
        person:
          max_area: 600000
          threshold: 0.8
          mask:
            - 50,1450,200,1450,200,1200,50,1200

  cam_6:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cam_6 # <--- the name here must match the name of the camera in restream
          roles:
            - record
            - detect
    detect:
      enabled: True
      width: 2688 # <---- update for your camera's resolution
      height: 1520 # <---- update for your camera's resolution
      fps: 5
      stationary:
        interval: 150
    record: # <----- Enable recording
      enabled: True
    snapshots: # <----- Enable snapshots
      enabled: True
    mqtt: # only apply to snapshots
      enabled: False
    zones:
      Worktop:
        coordinates: 1771,531,2688,865,2403,1520,1987,1520,1743,1520,2332,877,1654,599
      Island:
        coordinates: 954,726,1285,1022,1609,865,1207,654
    motion:
      mask:
        - 2600,60,2600,125,2000,125,2000,60
    objects:
      filters:
        person:
          max_area: 600000
          threshold: 0.8
          mask:
            - 2201,797,2288,849,2192,971,2688,1195,2688,714,2267,546
        cat:
          max_area: 80000

  cam_7:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cam_7 # <--- the name here must match the name of the camera in restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/cam_7_sub # <--- the name here must match the name of the camera_sub in restream
          roles:
            - detect
    record: # <----- Enable recording
      enabled: True
    snapshots: # <----- Enable snapshots
      enabled: True
    mqtt: # only apply to snapshots
      enabled: False
    objects:
      mask:
        - 0,0,0,409,43,498,220,374,184,0

  cam_8:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cam_8 # <--- the name here must match the name of the camera in restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/cam_8_sub # <--- the name here must match the name of the camera_sub in restream
          roles:
            - detect
    record: # <----- Enable recording
      enabled: True
    snapshots: # <----- Enable snapshots
      enabled: True
    mqtt: # only apply to snapshots
      enabled: False
    objects:
      filters:
        person:
          threshold: 0.81
        cat:
          max_area: 30000
          threshold: 0.8

  cam_9:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cam_9 # <--- the name here must match the name of the camera in restream
          roles:
            - record
            - detect
    detect:
      enabled: True
      width: 2688 # <---- update for your camera's resolution
      height: 1520 # <---- update for your camera's resolution
      fps: 5
      stationary:
        interval: 150
    record: # <----- Enable recording
      enabled: True
    snapshots: # <----- Enable snapshots
      enabled: True
    mqtt: # only apply to snapshots
      enabled: False
    motion:
      mask:
        - 426,29,426,60,80,61,80,27

Relevant log output

N/A

FFprobe output from your camera

ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[rtsp @ 0x5655369f9140] method SETUP failed: 461 Unsupported transport
Input #0, rtsp, from 'rtsp://127.0.0.1:8554/cam_1':
  Metadata:
    title           : go2rtc/1.6.0
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt470bg/bt470bg/bt709, progressive), 2688x1520, 15 fps, 15 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 8000 Hz, mono, fltp

Frigate stats

No response

Operating system

Proxmox

Install method

HassOS Addon

Coral version

USB

Network connection

Wired

Camera make and model

Dahua HDW5442TM

Any other information that may be helpful

No response

NickM-27 · 2024-02-23T22:46:54Z

NickM-27
Feb 23, 2024
Collaborator Sponsor

is frigate fully starting? there should be a .vacuum file in /config which will show when the database was last vacuumed. That is a very large database

12 replies

heisenberg2980 Feb 23, 2024
Author

I think this might be a bit over my basic docker knowledge as I have never made or run a custom dockerfile, but can I run this in the same Proxmox container where I am running Frigate using docker-compose?

heisenberg2980 Feb 24, 2024
Author

After querying the database I found out why the size of the database is so big, and the reason is that when I originally setup Frigate I decided to save the snapshots forever (well, 10000 days to be precise) as their size in the disk is insignificant compared to the video recordings (which obviously cannot be stored for so long due to storage limitations), and I just realised the events database includes a thumbnail of the events, increasing the record size to almost 10Kb, which makes the 200k events currently stored in my database to use around 2Gb.
Is there a reason to store this thumbnail in the database instead of storing it in the disk together with the other files (snapshots and videos)?

EDIT: answering my own question, the thumbnails are used in the UI to display multiple events at once, so I guess having to access them from disk would be too slow for the UI to be responsive

EDIT2: if this memory issue is caused by the size of the database as it seems to be, I guest a workaround could be to manually delete from the database all the events older than the maximum recording time (120 days in my case) so the snapshots will stay on the disk forever with the only down side that they will not be accessible from the Frigate UI

NickM-27 Feb 24, 2024
Collaborator Sponsor

The reason thumbnails are stored in the db is due to limitations for the home assistant integration. They have to be base64 encoded because the integration can not load from a file in the notification. There is a long thread over there about it but that is the gist best I can explain it.

heisenberg2980 Mar 31, 2024
Author

Small update about this issue: after creating a daily cron task to copy the thumbnails to a different folder (btw I decided to copy only the jpg version as the png one is 10x bigger and has similar quality) I updated the config file to save snapshots for only 120 days (same as events) and after the database was vacuumed it size has reduced from 3Gb to 1.7Gb, and the memory used by the container has also been reduced by a lot.

As a side note, maybe a future improvement could be for Frigate to store the clean version of the pictures in jpg format instead of png to save a lot of disk space.

NickM-27 Mar 31, 2024
Collaborator Sponsor

in the next version of frigate all new image files (as in new types of images needed for the new ui) will be stored as webp

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Support]: Frigate v0.13 consuming a lot more memory than v0.12 #10005

{{title}}

Replies: 1 comment 12 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[Support]: Frigate v0.13 consuming a lot more memory than v0.12 #10005

heisenberg2980 Feb 23, 2024

Describe the problem you are having

Version

Frigate config file

Relevant log output

FFprobe output from your camera

Frigate stats

Operating system

Install method

Coral version

Network connection

Camera make and model

Any other information that may be helpful

Replies: 1 comment · 12 replies

NickM-27 Feb 23, 2024 Collaborator Sponsor

heisenberg2980 Feb 23, 2024 Author

heisenberg2980 Feb 24, 2024 Author

NickM-27 Feb 24, 2024 Collaborator Sponsor

heisenberg2980 Mar 31, 2024 Author

NickM-27 Mar 31, 2024 Collaborator Sponsor

heisenberg2980
Feb 23, 2024

Replies: 1 comment 12 replies

NickM-27
Feb 23, 2024
Collaborator Sponsor

heisenberg2980 Feb 23, 2024
Author

heisenberg2980 Feb 24, 2024
Author

NickM-27 Feb 24, 2024
Collaborator Sponsor

heisenberg2980 Mar 31, 2024
Author

NickM-27 Mar 31, 2024
Collaborator Sponsor