[Detector Support]: Audio Detectors | Missed sounds

[Nsbx]

Describe the problem you are having

Hello,

I installed Frigate yesterday mainly for audio detection.

I'm having problems with some neighbors and need “proof” of their incivility.

I made recordings with my phone, but I wanted to make it automatic and more practical.

So I installed frigate, configured using the feeds I already had.

After a while, I had a working demo.

For calibration purposes, I activated all the event types to see which would be the most appropriate in my case.

I went to bed, but when I woke up, I was surprised to see that I had “only” five recordings and only 4 types.
“Cough”, “Fart”, “Snoring”, “Breathing”

And indeed, after opening my roller shutters, I noticed that no recordings had been made.

That despite the knocks I make on my desk, nothing is recorded either.

And even when I cough, 1 event in 5 is recorded.

Yet I've set the sensitivity to minimum.

The CPU and Memory figures are clearly normal.

Could the problem be with my camera?

The AI model used? (Is there a simple way to train the model with my own sounds to broaden the possibilities?)

Is it possible to simply record any noise above a certain number of decibels and categorize it as unknown?

Thanks in advance.

Version

0.13.2-6476f8a

Frigate config file

# config generated on 2024.05.20 08:43:15
mqtt:
  enabled: False

ffmpeg:
  output_args:
    record: preset-record-generic-audio-aac

go2rtc:
  streams:
    camera:
      - rtsp://192.168.1.31/ch0_1.h264
      - "ffmpeg:camera#audio=aac"
      
cameras:
  camera:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera
          roles:
            - record
            - audio
    detect: 
      enabled: False
    record:
      enabled: True
    audio:
      enabled: True
      min_volume: -100
      max_not_heard: 30
      listen:
        - babbling
        - yell
        - bellow
        - whoop
        - whispering
        - laughter
        - snicker
        - crying
        - sigh
        - singing
        - choir
        - sodeling
        - chant
        - mantra
        - child_singing
        - synthetic_singing
        - rapping
        - humming
        - groan
        - grunt
        - whistling
        - breathing
        - wheeze
        - snoring
        - gasp
        - pant
        - snort
        - cough
        - throat_clearing
        - sneeze
        - sniff
        - run
        - shuffle
        - footsteps
        - chewing
        - biting
        - gargling
        - stomach_rumble
        - burping
        - hiccup
        - fart
        - hands
        - finger_snapping
        - clapping
        - heartbeat
        - heart_murmur
        - cheering
        - applause
        - chatter
        - crowd
        - children_playing
        - animal
        - pets
        - dog
        - bark
        - yip
        - cat
        - purr
        - meow
        - caterwaul
        - cattle
        - gobble
        - honk
        - wild_animals
        - roaring_cats
        - roar
        - bird
        - pigeon
        - coo
        - crow
        - caw
        - owl
        - hoot
        - dogs
        - bell
        - church_bell
        - jingle_bell
        - bicycle_bell
        - wind
        - rustling_leaves
        - wind_noise
        - thunderstorm
        - thunder
        - water
        - rain
        - raindrop
        - rain_on_surface
        - stream
        - waterfall
        - ocean
        - waves
        - steam
        - gurgling
        - fire
        - crackle
        - vehicle
        - boat
        - sailboat
        - rowboat
        - motorboat
        - ship
        - motor_vehicle
        - car
        - toot
        - car_alarm
        - power_windows
        - skidding
        - tire_squeal
        - car_passing_by
        - race_car
        - truck
        - air_brake
        - air_horn
        - reversing_beeps
        - ice_cream_truck
        - bus
        - emergency_vehicle
        - police_car
        - ambulance
        - fire_engine
        - motorcycle
        - traffic_noise
        - rail_transport
        - train
        - train_whistle
        - train_horn
        - railroad_car
        - train_wheels_squealing
        - subway
        - aircraft
        - aircraft_engine
        - jet_engine
        - propeller
        - helicopter
        - fixed-wing_aircraft
        - bicycle
        - skateboard
        - engine
        - light_engine
        - dental_drill's_drill
        - lawn_mower
        - chainsaw
        - medium_engine
        - heavy_engine
        - engine_knocking
        - engine_starting
        - idling
        - accelerating
        - door
        - doorbell
        - ding-dong
        - sliding_door
        - slam
        - knock
        - tap
        - squeak
        - cupboard_open_or_close
        - drawer_open_or_close
        - dishes
        - cutlery
        - chopping
        - frying
        - microwave_oven
        - blender
        - water_tap
        - sink
        - bathtub
        - hair_dryer
        - toilet_flush
        - toothbrush
        - electric_toothbrush
        - vacuum_cleaner
        - zipper
        - keys_jangling
        - coin
        - scissors
        - electric_shaver
        - shuffling_cards
        - typing
        - typewriter
        - computer_keyboard
        - writing
        - alarm
        - telephone
        - telephone_bell_ringing
        - ringtone
        - telephone_dialing
        - dial_tone
        - busy_signal
        - alarm_clock
        - siren
        - civil_defense_siren
        - buzzer
        - smoke_detector
        - fire_alarm
        - foghorn
        - whistle
        - steam_whistle
        - mechanisms
        - ratchet
        - clock
        - tick
        - tick-tock
        - gears
        - pulleys
        - sewing_machine
        - mechanical_fan
        - air_conditioning
        - cash_register
        - printer
        - camera
        - single-lens_reflex_camera
        - tools
        - hammer
        - jackhammer
        - sawing
        - filing
        - sanding
        - power_tool
        - drill
        - explosion
        - gunshot
        - machine_gun
        - fusillade
        - artillery_fire
        - cap_gun
        - fireworks
        - firecracker
        - burst
        - eruption
        - boom
        - wood
        - chop
        - splinter
        - crack
        - glass
        - chink
        - shatter
        - liquid
        - splash
        - slosh
        - squish
        - drip
        - pour
        - trickle
        - gush
        - fill
        - spray
        - pump
        - stir
        - boiling
        - sonar
        - arrow
        - whoosh
        - thump
        - thunk
        - electronic_tuner
        - effects_unit
        - chorus_effect
        - basketball_bounce
        - bang
        - slap
        - whack
        - smash
        - breaking
        - bouncing
        - whip
        - flap
        - scratch
        - scrape
        - rub
        - roll
        - crushing
        - crumpling
        - tearing
        - beep
        - ping
        - ding
        - clang
        - squeal
        - creak
        - rustle
        - whir
        - clatter
        - sizzle
        - clicking
        - clickety-clack
        - rumble
        - plop
        - hum
        - zing
        - boing
        - crunch
        - sine_wave
        - harmonic
        - chirp_tone
        - sound_effect
        - pulse
        - inside
        - outside
        - reverberation
        - echo
        - noise
        - environmental_noise
        - static
        - mains_hum
        - distortion
        - sidetone
        - cacophony
        - white_noise
        - pink_noise
        - throbbing
        - vibration
        - television
        - radio
        - field_recording

docker-compose file or Docker CLI command

name: frigate
services:
  frigate:
    cpu_shares: 90
    command: []
    container_name: frigate
    deploy:
      resources:
        limits:
          memory: 28989M
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/apex_0:/dev/apex_0
      - /dev/dri/renderD128:undefined
    entrypoint:
      - bash
      - -c
      - >-
        if [ ! -f "/config/config.yml" ]; then
          echo -n "No config file found. Create one... "
          {
            echo "# config generated on $(date +%Y.%m.%d\ %H:%M:%S)"
            echo "mqtt:"
            echo "  enabled: False"
            echo ""
            echo "cameras:"
            echo "  sample_camera: # <------ Name the camera"
            echo "    ffmpeg:"
            echo "      inputs:"
            echo "        - path: rtsp://192.168.255.10/stream # <----- The stream you want to use for detection"
            echo "          roles:"
            echo "            - detect"
            echo "    detect:"
            echo "      enabled: False # <---- disable detection until you have a working camera feed"
            echo "      width: 1280 # <---- update for your camera resolution"
            echo "      height: 720 # <---- update for your camera resolution"
          } > /config/config.yml
          if [ $? -eq 0 ]; then
            echo "done."
          else
            echo "error."
            exit 1
          fi
        fi

        /init
    environment:
      - TZ=Europe/Paris
    hostname: frigate
    healthcheck:
      test:
        - CMD-SHELL
        - curl --fail -s http://localhost:5000/ || exit 1
      timeout: 10s
      interval: 1m30s
      retries: 3
    image: ghcr.io/blakeblackshear/frigate:stable
    labels:
      icon: https://raw.githubusercontent.com/mr-manuel/CasaOS-HomeAutomation-AppStore/latest/Apps/Frigate/icon.png
    ports:
      - target: 5000
        published: "5000"
        protocol: tcp
      - target: 8554
        published: "8554"
        protocol: tcp
      - target: 8555
        published: "8555"
        protocol: tcp
      - target: 8555
        published: "8555"
        protocol: udp
    privileged: true
    restart: unless-stopped
    shm_size: "67108864"
    volumes:
      - type: bind
        source: /etc/localtime
        target: /etc/localtime
        read_only: true
      - type: bind
        source: /DATA/AppData/frigate/config
        target: /config
      - type: bind
        source: /DATA/AppData/frigate/media
        target: /media/frigate
    x-casaos:
      ports:
        - container: "5000"
          description:
            en_us: Web GUI port
        - container: "8554"
          description:
            en_us: RTSP feeds
        - container: "8555"
          description:
            en_us: WebRTC over tcp and udp
      volumes:
        - container: /config
          description:
            en_us: Contains all configuration files.
        - container: /media/frigate
          description:
            en_us: Contains database and all media files.
    cap_add: []
    network_mode: bridge
x-casaos:
  architectures:
    - amd64
    - arm64
    - arm
  author: mr-manuel
  category: Surveillance
  description:
    en_us: >
      Use of a Google Coral Accelerator is optional, but highly recommended. The
      Coral will outperform even the best CPUs and can process 100+ FPS with
      very little overhead.


      * Tight integration with Home Assistant via a custom component

      * Designed to minimize resource use and maximize performance by only looking for objects when and where it is necessary

      * Leverages multiprocessing heavily with an emphasis on realtime over processing every frame

      * Uses a very low overhead motion detection to determine where to run object detection

      * Object detection with TensorFlow runs in separate processes for maximum FPS

      * Communicates over MQTT for easy integration into other systems

      * Records video with retention settings based on detected objects

      * 24/7 recording

      * Re-streaming via RTSP to reduce the number of connections to your camera

      * WebRTC & MSE support for low-latency live view


      Visit [Frigate](https://frigate.video/) for more details. Check the [documentation](https://docs.frigate.video/) for further settings.
  developer: Blake Blackshear
  hostname: ""
  icon: https://raw.githubusercontent.com/mr-manuel/CasaOS-HomeAutomation-AppStore/latest/Apps/Frigate/icon.png
  index: /
  is_uncontrolled: false
  main: frigate
  port_map: "5000"
  project_url: https://github.com/mr-manuel/CasaOS-HomeAutomation-AppStore
  scheme: http
  screenshot_link:
    - https://raw.githubusercontent.com/mr-manuel/CasaOS-HomeAutomation-AppStore/latest/Apps/Frigate/screenshot-1.png
    - https://raw.githubusercontent.com/mr-manuel/CasaOS-HomeAutomation-AppStore/latest/Apps/Frigate/screenshot-2.png
    - https://raw.githubusercontent.com/mr-manuel/CasaOS-HomeAutomation-AppStore/latest/Apps/Frigate/screenshot-3.png
    - https://raw.githubusercontent.com/mr-manuel/CasaOS-HomeAutomation-AppStore/latest/Apps/Frigate/screenshot-4.png
    - https://raw.githubusercontent.com/mr-manuel/CasaOS-HomeAutomation-AppStore/latest/Apps/Frigate/screenshot-5.png
    - https://raw.githubusercontent.com/mr-manuel/CasaOS-HomeAutomation-AppStore/latest/Apps/Frigate/screenshot-6.png
    - https://raw.githubusercontent.com/mr-manuel/CasaOS-HomeAutomation-AppStore/latest/Apps/Frigate/screenshot-7.png
  store_app_id: frigate
  tagline:
    en_us: >
      A complete and local NVR designed for Home Assistant with AI object
      detection. Uses OpenCV and Tensorflow to perform realtime object detection
      locally for IP cameras.
  thumbnail: https://raw.githubusercontent.com/mr-manuel/CasaOS-HomeAutomation-AppStore/latest/Apps/Frigate/icon.png
  tips:
    before_install:
      en_us: >
        Frigate will use ports `5000/tcp`, `8554/tcp`, `8555/tcp` and
        `8555/udp`. After installing further setup is needed.


        [Website](https://frigate.video) &bull; [Documentation](https://docs.frigate.video/guides/getting_started)
  title:
    custom: ""
    en_us: Frigate NVR

Relevant log output

No Relevant log output

Operating system

Debian

Install method

Docker Compose

Coral version

CPU (no coral)

Any other information that may be helpful

No response

[NickM-27]

They're probably just not scoring high enough.

[heisenberg2980]

@NickM-27 I have the same issue. Trying to recognise fireworks sound, and even after lowering the threshold to 0.5 and playing youtube videos with fireworks at a very high volume, frigate doesn´t recognise the sound as fireworks. I did the same test with the object piano with the same results.

It seems Frigate is only able to pick some of the sounds (because it works with objects like music, speech, and cat) but not all of them, is there anything we can do to debug this issue? Maybe the threshold value is not being applied to sound detection?

[NickM-27]

It's most likely just the model but detecting it with high confidence, YouTube videos aren't always the best test and we've seen some users say it depends on the sound. Like someone had two dogs and one always triggered bark while the other rarely did

[heisenberg2980]

@NickM-27 I wish that was the case, but I have tried multiple videos from different platforms and none of them trigger the fireworks recognition. The test with the piano is even easier to test, because when playing a piano frigate recognizes the object music inmediately, but it doesn´t recognize the object piano, so it´s clearly something going on there. You can do the test yourself, and you will see how your frigate doesn´t recognize a piano, no matter what you do.

For video we have the debug screen where we can "see" what frigate is recognizing, but is there any way to debug issues with the audio recognition?