[Config Support]: h264 recording with h265 cameras

[aleare]

Describe the problem you are having

Hi, I'm using a Beelink EQ14 (Intel N100) with a coral USB accelerator. Frigate is running in docker on a Debian 12 bare-metal system. I use frigate to interface with a Reolink NVR that hosts 12 Reolink RLC 811A 4k cameras. I also have HA running on HAos on a RaspberryPi that acts as a MQTT broker and interfaces with Frigate via its integration.

After many tries, I've arrived at a configuration that works and gives no error in Frigate. This configuration uses the RLC 811A's main RTSP h265 stream for recording and the RTMP "ext" stream for detection. I have attached the redacted version of this config below.

On a computer everything is fine, unfortunately I discovered that mobile phones do not have the h265 decode capability so when I get a notification to my mobile I cannot see the video snippet of the event. I've seen on this Github discussions that you can use frigate to convert h265 to h264 for the recordings (even if it's not efficient). I've changed my configuration for all cameras as so:

go2rtc:
  streams:
    cancello_sin:
      - rtsp://<user>:<pass>@<ip>:554/h265Preview_01_main
      - ffmpeg:cancello_sin#video=h264#hardware#audio=opus
...
cameras:
  cancello_sin:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cancello_sin?video=copy&audio=opus
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: 
            https://<ip>/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=<user>&password=<pass>
          input_args: preset-rtmp-generic
...

However I still get clips encoded as h265 and so not viewable on mobiles. I also noticed that the IGpu is always at 0% so it's clear this is not working. I've experimented in changing both the go2rtc and the cameras\ffmpeg part of the configuration but I either get errors and no streams or recordings in h265.

I've tried looking for a solution here and on reddit but to no avail so I'm at a loss. Can somebody please suggests me what I'm doing wrong ?

Version

0.14.1-f4f3cfa

Frigate config file

mqtt:
  enabled: true
  port: <>
  host: <>
  user: <>
  password: <>

detect:
  fps: 10

detectors:
  coral:
    type: edgetpu
    device: usb

objects:
  track:
    - person
    - car
    - dog
    - cat

record:
  enabled: true
  retain:
    days: 1
    mode: active_objects
  events:
    retain:
      default: 30
      mode: motion

ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    record: preset-record-generic-audio-copy

review:
  alerts:
    # Optional: labels that qualify as an alert
    labels:
      - car
      - person
  detections:
    # Optional: labels that qualify as a detection
    labels:
      - car
      - person

snapshots:
  # Optional: Enable writing jpg snapshot to /media/frigate/clips
  enabled: true
  retain:
    default: 10
    objects:
      person: 15
  quality: 100

go2rtc:
  streams:
    cancello_sin:
      - rtsp://<user>:<pass>@<ip>:554/h265Preview_01_main
      - ffmpeg:cancello_sin#audio=opus
    cancello_des:
      - rtsp://<user>:<pass>@<ip>:554/h265Preview_02_main
      - ffmpeg:cancello_des#audio=opus
    ...

cameras:
  cancello_sin:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cancello_sin?video=copy&audio=opus
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: 
            https://<ip>/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=<user>&password=<pass>
          input_args: preset-rtmp-generic
          roles:
            - detect
    ...
version: 0.14

Relevant Frigate log output

No relevant Frigate error logs

Relevant go2rtc log output

No relevant go2rtc logs

Frigate stats

No response

Operating system

Debian

Install method

Docker Compose

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    restart: unless-stopped
    shm_size: "104mb"
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/dri/card0:/dev/dri/card0
    image: ghcr.io/blakeblackshear/frigate:stable
    cap_add:
      - CAP_PERFMON
    privileged: true
    volumes:
      - ./config:/config
      - ./storage:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp

Object Detector

Coral

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

I've also thought that maybe the IGpu was the problem so trying different things I found out that:

• Cannot seem to get the i965 driver to work with the Debian 12 (oldish) linux firmware (VaInfo: libva error: /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so init failed).
• Cannot seem to use Quick sync with h265 streams in frigate (black feed, errors) (maybe related to this: [HW Accel Support]: QSV does not work with H265 #8989 ?).

[NickM-27]

unfortunately I discovered that mobile phones do not have the h265 decode capability

In general that's not true, the vast majority of phones support h265 as long as the browser used supports h265

[aleare]

All my family (unfortunately) uses Iphones (safari). As I said, on computers the video works but on the phones no, so doing a google search I found that the h265 was the issue for the impossible reproduction. Even on video testing websites, trying to load a h265 on an Iphone gives the same error\problem:

[aleare]

Update: I found on a frigate-unrelated subreddit (https://www.reddit.com/r/MacOS/comments/172b2jn/h265_browser_compatibility) this comment:

If you're encoding the vids with FFMPEG, don't forget to include the -tag:v hvc1 (source). This fixed my issue playing x265 videos directly in Safari for Mac/iOS. I can't speak for other encoders though.

and going into the FFMPEG docs (https://trac.ffmpeg.org/wiki/Encode/H.265#FinalCutandApplestuffcompatibility) this:

To make your file compatible with Apple "industry standard" H.265 you have to add the following argument -tag:v hvc1

So maybe I just need to pass this parameter\flag (I have no idea what it does) on the FFMPEG inside frigate? But I don't know how to do that or even if it is possible...

[NickM-27]

See this example https://docs.frigate.video/configuration/camera_specific#annke-c800

[aleare]

The example uses:
record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -tag:v hvc1 -bsf:v hevc_mp4toannexb -c:a aac
That configuration lead to errors\black feed. After many different tries I found out it was caused by the audio encoder. So finally I arrived at this configuration that works perfectly with my reolink h265 cameras and correctly displays h265 on iphones:

ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime
      1 -c copy -tag:v hvc1 -bsf:v hevc_mp4toannexb
...
go2rtc:
  streams:
    camera_name:
      - rtsp://<user>:<pass>@<ip>/h265Preview_01_main
      - ffmpeg:camera_name
...
cameras:
  camera_name:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera_name?video=copy
          input_args: preset-rtsp-restream
          roles:
            - record
...

If you don't wanna have audio at all use instead:
record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -an -tag:v hvc1 -bsf:v hevc_mp4toannexb

Since Iphones are very common and this problem affects all of the user of h265 cameras, I think a more detailed (and Annke C800 unrelated) paragraph should be added to the documentation or alternatively added as a output args preset.
Finally, thanks a lot @NickM-27 !

Ps: While troubleshooting i found out another small issue, but since it is totally unrelated to this one I think should open a different discussion.

[NickM-27]

Since Iphones are very common and this problem affects all of the user of h265 cameras,

most cameras already work this way and this config is not needed, hence why it is camera specific.

[aleare]

Well then at least add it to the Reolink camera specific config since I would have never looked at the Annke config to fix an issue of my Reolink cameras :).

Also do you mean that most h265 cameras work out of the box with hvc1 encoding or that simply most cameras use h264? Since if it is the second then considering h265 is much more efficient with higher quality at lower bitrate, then more and more cameras in the future will use h265.

Anyway that's just a suggestion, I'm not a contributor of the project but in general I find the documentation unnecessarily too succinct and lacking on many parts that could reduce the number of discussions here and on reddit. (For example to understand what the preset-rtsp-generic did, that worked fine with my previous config, I had to go into the source files here on github and check the ffmpeg arguments that corresponded to that preset: that's a bit 'too much' for a less tech savvy user)

[NickM-27]

High quality cameras that support h265 do not need this fix. In Frigate 0.15 there will be an additional ffmpeg config that applies this fix and it is documented

[aleare]

Great to hear, 0.15 is shaping up to be a major Frigate gamechanger!