[Camera Support]: Dahua DH-IPC-HDW2531EMP-AS-S2 autotracking

[therokh]

Describe the problem you are having

Hi, I have a Dahua DH-IPC-HDW2531EMP-AS-S2 PTZ camera and have a mostly working autotracking setup. I have noticed though that often the tracker will detect a person, pan/tilt and then zoom to them and in the time it takes to zoom, the person moves out of frame and the tracker loses them. I have relative zooming enabled but it seems like it's doing a separate zoom movement. I've tried changing the zoom_factor and can't say that it made a noticeable difference. The video attached shows the behaviour.

The other issue is that the tracker often over-corrects on close objects and moves them out of frame, and then loses them. I've tried with and without movement weights and I can't say I've noticed it being any better. This happens mostly with people walking by a path at the bottom of frame, which is probably 5m away from the camera.

Version

0.15.0-33825f6

What browser(s) are you using?

No response

Frigate config file

cameras:  
  front_ptz:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front_ptz?video&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/front_ptz_small
          input_args: preset-rtsp-restream
          # Use smaller resolution stream for detection and thumbnails.
          roles:
            - detect
    onvif:
      host: 10.3.8.4
      port: 80
      user: redact
      password: redact
      autotracking:
        calibrate_on_startup: false
        enabled: true
        zooming: relative
        zoom_factor: 0.4
        track:
          - person
          - cat
        required_zones:
          - ptz_opposite_path
          - ptz_footpath
          - ptz_road
        return_preset: home
        #movement_weights: 0.25, 1.0, 7.361041784286499, -287242838.3571313, 646077143.064353
        #movement_weights: 0.25, 1.0, 3.683077335357666, 4.11272554961584, 0.1503896736329604
        #movement_weights: 0.25, 1.0, 3.496250867843628, 3.6256675453596214, 0.7419491637137641
    objects:
      track:
        - person
        - dog
        - cat
        - bird
        - car
      filters:
        car:
          mask: 0,0.813,0.315,0.809,0.405,0.994,0,1
    detect:
      width: 706
      height: 576
    review:
      alerts:
        required_zones:
          - ptz_footpath
          - ptz_road_parking
      detections:
        required_zones:
          - ptz_road
          - ptz_opposite_path
    zones:
      ptz_footpath:
        coordinates: 1,0.835,0.801,0.923,0.429,0.981,0,0.964,0,0.67,0.501,0.623,1,0.537
        inertia: 3
        loitering_time: 1
      ptz_road:
        coordinates: 0,0.403,0.415,0.315,0.688,0.297,1,0.304,0.998,0.521,0.495,0.599,0,0.644
        loitering_time: 0
        inertia: 3
        objects:
          - bird
          - cat
          - dog
          - person
      ptz_opposite_path:
        coordinates: 1,0.166,0.495,0.193,0.115,0.248,0.017,0.365,0.02,0.401,0.537,0.303,1,0.295
        inertia: 3
        loitering_time: 0
        objects:
          - bird
          - cat
          - dog
          - person
      ptz_road_parking:
        coordinates: 0.436,0.609,0.435,0.509,0.689,0.471,1,0.44,0.992,0.518,1,0.571,0.433,0.659
        loitering_time: 5
        inertia: 3
        objects: car
    motion:
      mask:
        - 0,0.731,0.408,0.784,0.441,1,0,1
        - 0.751,0.93,0.811,1,1,1,1,0.815,0.859,0.842
        - 0,0.268,0.11,0.198,0.114,0.3,0.241,0.26,0.229,0.157,0.396,0.093,0.375,0,0,0
        - 0.323,0,0.354,0.141,0.491,0.125,0.502,0.186,0.526,0.181,0.544,0.093,0.589,0.119,0.611,0.182,0.811,0.233,0.819,0.148,0.883,0.136,0.955,0.231,1,0.322,1,0

Relevant Frigate log output

N/A

Relevant go2rtc log output

N/A

FFprobe output from your camera

ffprobe version 7.0.2 Copyright (c) 2007-2024 the FFmpeg developers
  built with gcc 14 (GCC)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 ' --extra-cflags=' -I/usr/include/rav1e' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --enable-chromaprint --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-lcms2 --enable-libaom --enable-libaribb24 --enable-libaribcaption --enable-libdav1d --enable-libass --enable-libbluray --enable-libbs2b --enable-libcodec2 --enable-libcdio --enable-libdrm --enable-libjack --enable-libjxl --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libharfbuzz --enable-libilbc --enable-libmp3lame --enable-libmysofa --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-libplacebo --enable-librsvg --enable-librav1e --enable-librubberband --enable-libqrencode --enable-libsmbclient --enable-version3 --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-version3 --enable-vapoursynth --enable-libvpx --enable-vulkan --enable-libshaderc --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-avfilter --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-lto --enable-libvpl --enable-runtime-cpudetect
  libavutil      59.  8.100 / 59.  8.100
  libavcodec     61.  3.100 / 61.  3.100
  libavformat    61.  1.100 / 61.  1.100
  libavdevice    61.  1.100 / 61.  1.100
  libavfilter    10.  1.100 / 10.  1.100
  libswscale      8.  1.100 /  8.  1.100
  libswresample   5.  1.100 /  5.  1.100
  libpostproc    58.  1.100 / 58.  1.100
Input #0, rtsp, from 'rtsp://redact:[email protected]:554/cam/realmonitor?channel=1&subtype=0':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.080000, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(tv, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn

Frigate stats

No response

Operating system

Other Linux

Install method

Docker Compose

Object Detector

OpenVino

Network connection

Wired

Camera make and model

Dahua DH-IPC-HDW2531EMP-AS-S2

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

front_ptz_r367gy.mp4

[hawkeye217]

Calibration is designed to help with scenarios like this, so you should definitely be running with movement_weights defined in your config. The movement_weights are particular to your camera's motor speeds. When they are set and an object enters the frame, Frigate calculates the object's speed and autotracking will move in anticipation of where the object may be after camera movement is complete. Without movement_weights defined, this "anticipation movement" doesn't happen.

The amount of time it takes to zoom is not part of this calculation though, which is a little problematic in your case because your initial zoom takes a fair amount of time.

There are a few things you can do to help this. You'll want to set up Frigate to track a person and have autotracking lock on to it sooner. This could mean things like moving your zone to trigger autotracking closer to the edge of the frame or changing your min_area or threshold (depending on how a person is scoring). Lowering your zoom_factor will also prevent Frigate from zooming in as much. Switching to absolute zooming should also prevent the initial zoom-in.

As for objects close to the camera, there's really not a lot that can be done to improve it because an object is moving very quickly across the frame relative to an object that is further away. There is too much lag when sending ONVIF movement commands to the camera. This is why Frigate's autotracking will never be able to perform well for objects close to the frame like autotracking implemented in a camera's firmware.

[hawkeye217]

I should also note that relative zooming has moved the pan/tilt and zoom motor together on the PTZs I have tested as I was writing the autotracking code. In your example, it does look like the firmware is moving them individually. The ONVIF standard sets no condition that they must move simultaneously, so I suppose this is just firmware-specific.

[hawkeye217]

As far as the movement_weights in your config that are commented out, the first one looks bogus. The second and third look similar to values I've seen.

        #movement_weights: 0.25, 1.0, 7.361041784286499, -287242838.3571313, 646077143.064353    <---- bogus values
        #movement_weights: 0.25, 1.0, 3.683077335357666, 4.11272554961584, 0.1503896736329604    <---- looks good
        #movement_weights: 0.25, 1.0, 3.496250867843628, 3.6256675453596214, 0.7419491637137641    <---- looks good

Those bogus values will almost certainly cause issues when autotracking, so I'd uncomment the 2nd or 3rd one and go with that.

[therokh]

Hey thanks for getting back so quickly. I've re-calibrated and have this now:

#movement_weights: 0.25, 1.0, 7.361041784286499, -287242838.3571313, 646077143.064353
#movement_weights: 0.25, 1.0, 3.683077335357666, 4.11272554961584, 0.1503896736329604
#movement_weights: 0.25, 1.0, 3.496250867843628, 3.6256675453596214, 0.7419491637137641
movement_weights: 0.25, 1.0, 3.749577760696411, 4.412547773955971, 1.4113262068840775

My zoning setup for the camera is currently this:

Trying to prevent motion trigger from the plant movement or reflections from the car. I find that objects in the ptz_opposite_path zone are generally tracked pretty well all the way up and down the street.
I'll leave these new weights in and see how it goes for a few days.

Here's a clip of the close up movement before adding the weights.
https://github.com/user-attachments/assets/c07eac1d-9e39-4aad-9312-008f91385c8f

[hawkeye217]

Motion masks on a PTZ are not ideal because the frame changes when the camera moves. If an object you're tracking ends up in a motion masked area, this may negatively affect Frigate's ability to continue tracking it. On my PTZ, the only thing I have masked is the timestamp. Your masks are not too close to the center of the frame, though - so you may be okay. Hard to know without watching debug view while it's actively autotracking.

Calibration should certainly help with objects in the footpath zone.

[therokh]

There's a bit of weirdness still with some of the movements. Not sure why it swung down to the car. Is there any option to include the debug bounding boxes in a recording?

front.mp4

[hawkeye217]

I'll need to see a screen capture of debug view in order to explain this behavior. There are a number of factors that could be contributing.

Could you also post your full frigate config and screenshots of the system metrics and camera metrics pages that follow a "ghost" tracking session?

[therokh]

Hey I managed to capture it with screen recorder just after the person was initially detected.

Screencast_20241127_092432.mp4

Full config:

database:
  path: /media/frigate/database/frigate.db

mqtt:
  host: redact
  user: redact
  password: redact

record:
  enabled: true
  retain:
    days: 60
    mode: motion
  alerts:
    retain:
      days: 90
  detections:
    retain:
      days: 90
detectors:
  ov:
    type: openvino
    device: AUTO

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  path: /openvino-model/ssdlite_mobilenet_v2.xml
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

# Use go2rtc for live previews.
go2rtc:
  streams:
    front_garage:
      - ffmpeg:rtsp://10.3.8.1:554/s0
      - ffmpeg:front_garage#audio=opus
    front_garage_small:
      - ffmpeg:rtsp://10.3.8.1:554/s2
    front_ptz:
      - ffmpeg:rtsp://redact:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    front_ptz_small:
      - rtsp://redact:[email protected]:554/cam/realmonitor?channel=1&subtype=1
    rear_garden:
      - ffmpeg:rtsp://redact:[email protected]:554/cam/realmonitor?channel=1&subtype=0
      - ffmpeg:front_garage#audio=opus
    rear_garden_small:
      - rtsp://redact:[email protected]:554/cam/realmonitor?channel=1&subtype=1

birdseye:
  enabled: true
  mode: continuous

ffmpeg:
  # hwaccel_args: preset-intel-qsv-h264
  hwaccel_args: preset-vaapi

cameras:
  # Unifi G3 Flex ========================================================================
  front_garage:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-aac
      inputs:
      # Stream from the go2rtc input stream instead of opening a second connection to camera.
        - path: rtsp://127.0.0.1:8554/front_garage
          input_args: preset-rtsp-generic
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/front_garage_small
          input_args: preset-rtsp-restream
        # Use smaller resolution stream for detection and thumbnails.
          roles:
            - detect
    objects:
      track:
        - person
        - dog
        - cat
        - bicycle
        - bird
    detect:
      width: 1024
      height: 576
    motion:
      mask:
        # Hedge
        - 0.159,1,0.125,0.61,0,0.495,0,1
        - 0.155,0,0.156,0.042,0.1,0.036,0.099,0.076,0,0.076,0,0
        - 0.91,0.036,0.93,0,0.762,0,0.665,0,0.685,0.036
    zones:
      driveway: # Driveway
        coordinates: 0.671,0.429,1,0.657,1,1,0.348,1,0.234,0.652
        inertia: 3
      mulch: # Mulch walkway
        coordinates: 0.159,1,0.346,1,0.236,0.655,0.13,0.7

        inertia: 3
      Front_road_parking:
        coordinates: 0.508,0.307,0.797,0.214,0.699,0.146,0.422,0.227
        loitering_time: 5
        objects: bicycle
      Front_left_path:
        coordinates: 0.004,0.407,0.175,0.327,0.293,0.62,0.139,0.695,0.123,0.609,0.001,0.493
        loitering_time: 0
        inertia: 3
      front_path:
        coordinates: 0.442,0.342,0.551,0.486,0.907,0.322,0.749,0.233
        loitering_time: 0
    review:
      alerts:
        required_zones:
          - mulch
          - driveway
      detections:
        required_zones:
          - mulch
          - driveway

  # Dahua PTZ blah =========================================================================
  front_ptz:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front_ptz?video&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/front_ptz_small
          input_args: preset-rtsp-restream
          # Use smaller resolution stream for detection and thumbnails.
          roles:
            - detect
    onvif:
      host: 10.3.8.4
      port: 80
      user: redact
      password: redact
      autotracking:
        calibrate_on_startup: false
        enabled: true
        zooming: relative
        zoom_factor: 0.4
        track:
          - person
          - cat
        required_zones:
          - ptz_opposite_path
          - ptz_footpath
          - ptz_road
        return_preset: home
        # movement_weights: 0.25, 1.0, 4.142112970352173, 4.890480953647245, 0.3357865748866921 # untested
        movement_weights: 0.25, 1.0, 3.749577760696411, 4.412547773955971, 1.4113262068840775 # working well?
    objects:
      track:
        - person
        - dog
        - cat
        - bird
        - car
      filters:
        car: {}
    detect:
      width: 706
      height: 576
      annotation_offset: -1250
    review:
      alerts:
        required_zones:
          - ptz_footpath
          - ptz_road_parking
      detections:
        required_zones:
          - ptz_road
          - ptz_opposite_path
    zones:
      ptz_footpath:
        coordinates: 1,0.835,0.801,0.923,0.429,0.981,0,0.964,0,0.67,0.501,0.623,1,0.537
        inertia: 3
        loitering_time: 1
      ptz_road:
        coordinates: 0,0.436,0.395,0.339,0.679,0.308,1,0.319,0.998,0.521,0.495,0.599,0,0.644
        loitering_time: 0
        inertia: 3
        objects:
          - bird
          - cat
          - dog
          - person
      ptz_opposite_path:
        coordinates: 1,0.166,0.495,0.193,0.115,0.248,0.017,0.365,0.012,0.431,0.475,0.327,1,0.31
        inertia: 3
        loitering_time: 0
        objects:
          - bird
          - cat
          - dog
          - person
      ptz_road_parking:
        coordinates: 0.436,0.609,0.435,0.509,0.689,0.471,1,0.44,0.992,0.518,1,0.571,0.433,0.659
        loitering_time: 5
        inertia: 3
        objects: car
    motion:
      mask: 0,0.064,0.437,0.058,0.437,0,0,0

  rear_garden:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-aac
      inputs:
      # Stream from the go2rtc input stream instead of opening a second connection to camera.
        - path: rtsp://127.0.0.1:8554/rear_garden
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/rear_garden_small
          input_args: preset-rtsp-restream
        # Use smaller resolution stream for detection and thumbnails.
          roles:
            - detect
    record:
      retain:
        days: 7
        mode: motion
      alerts:
        retain:
          days: 30
      detections:
        retain:
          days: 30
    objects:
      track:
        - person
        - dog
        - cat
        - bird
    detect:
      width: 704
      height: 576
    motion:
      mask:
        # Timestamp
        - 0,0,0.433,0,0.432,0.079,0,0.084
        - 0,0.914,0.07,0.927,0.297,0.836,0.503,0.718,0.622,0.528,0.786,0.381,0.907,0.265,0.933,0.238,0.948,0,0.732,0,0.459,0,0.296,0,0.288,0,0,0
    zones:
      Rear_Yard:
        coordinates: 
          0.938,0.449,0.977,0.528,0.883,0.843,0.8,0.999,0.189,0.999,0.153,0.918,0.457,0.714,0.933,0.242,0.988,0.278,0.966,0.343
        loitering_time: 0
        inertia: 3

version: 0.15-0
camera_groups:
  Front:
    order: 1
    icon: LuWarehouse
    cameras:
      - front_garage
      - front_ptz
semantic_search:
  enabled: true
  reindex: false
  model_size: small

[hawkeye217]

Thanks. Does this happen every time with every person in that zone, or is it just occasional?

Does it still happen if you disable zooming?

[therokh]

It's occasional, and I have usually only noticed it for people up close, but in this case it was across the road. And I feel like it seems to happen mostly for people moving from right to left.

I will disable zooming and see if it keeps happening.

[hawkeye217]

It would be helpful to see debug logs during one of these ghost events as well. Be advised - the logs are quite verbose with autotracking debugging enabled.

logger:
  default: info
    frigate.ptz.autotrack: debug
    frigate.ptz.onvif: debug