Gradually increasing video & audio desync over period of a stream

[morgant]

A long-standing known issue that I've experienced is video & audio gradually becoming further out of sync over the duration of a stream. The longer the stream, the worse it is by the end. I've found this is less of an issue on faster hardware, but still exists.

Recently, I've been testing renice(8)-ing fauxstream, sndiod, and ffmpeg processes upon starting my stream to ensure they have the highest priority, e.g. renice -n -20 <pid>. Of course, this must be done as the superuser, which I don't want to run fauxstream itself as. Per the manual:

Users other than the superuser may only alter the priority of processes they own, and can only monotonically increase their “nice value” within the range 0 to PRIO_MAX (20), which prevents overriding administrative fiats. The superuser may alter the priority of any process and set the priority to any value in the range PRIO_MIN (-20) to PRIO_MAX.

This may help, but @rfht did some testing in a stream yesterday -- which I unfortunately missed -- and mentioned the following in #openbsd-gaming on Libera.chat:

it looks like using -max_interleave_delta 50000 helps with keeping things in sync

Per the FFmpeg Formats Documentation:

max_interleave_delta integer (output)

Set maximum buffering duration for interleaving. The duration is expressed in microseconds, and defaults to 10000000 (10 seconds).

To ensure all the streams are interleaved correctly, libavformat will wait until it has at least one packet for each stream before actually writing any packets to the output file. When some streams are "sparse" (i.e. there are large gaps between successive packets), this can result in excessive buffering.

This field specifies the maximum difference between the timestamps of the first and the last packet in the muxing queue, above which libavformat will output a packet regardless of whether it has queued a packet for all the streams.

If set to 0, libavformat will continue buffering packets until it has a packet for each stream, regardless of the maximum timestamp difference between the buffered packets.

So, I'll definitely give that a try too on my next stream.

[morgant]

@rfht has been testing the following, which uses VA-API, plus some performance & desync improvements:

https://gist.github.com/rfht/153388d836bcc7235b516a4216421d79

[morgant]

@rfht I recall you mentioning on #openbsd-gaming that you were still experiencing desync on your streams, even with this solution. It was quite some time ago and I haven't had a chance to test this implementation yet. 😦

I can't remember, do you override sndiod(8)'s -b option for lower-latency audio? For example, the OpenBSD FAQ's Multimedia section suggests 50ms for gaming.

Further thoughts, looking through the ffmpeg documentation again:

• Maybe we could use loopback decoders to handle the video and both audio (monitor & mic) in a single ffmpeg command?
• If we can handle it all in one command and remove piping entirely, would we then be able to utilize the -isync option on one of the streams (maybe video?) to help ffmpeg keep things synced?

I'll try to experiment with this soon.

---

Further note-to-self: I mentioned in this issue's description that I've been using renice(8). I wanted to note that sndiod(8) is already nice(1)-ed to -20, so any renice(8)-ing should probably use a lower value. obsdfreqd(1) uses -15, so I think a similar value (maybe even -10) would be appropriate for ffmpeg and/or fauxstream.