Speaker Networks

[fatboychummy]

Context

I've been toying around with audio and speakers in CC a lot lately, and a major issue I've noticed is desync, especially if you have a large amount of speakers. Even some of the current best libraries for working with audio (like JackMac's AUKit) are riddled with issues of desync -- and I don't think it's really that easy for them to fix it. The fact that the audio is played on the clients means that syncing multiple speakers is not going to be easy when each speaker doesn't know about each other speaker and what they're currently playing.

So the issue now becomes, how do we fix this issue without making it seem unrealistic, or without having to overhaul everything and breaking back compat? If we can't solve it in software, it will need to be solved with 'hardware.'

Suggestion

Speaker networks, like the title of the feature request.

I believe Computronics really had a good route. One block acted as a 'controller' of sorts (Tape Drive), while the actual speakers themselves connected to it to play audio. No, I'm not saying to add in tape drives, but having one block control multiple speakers may be the way to go about this.

One peripheral call to play audio on all connected speakers, with a separate "audio cable" to connect speakers (or perhaps it just plays on all connected speakers on the same CC network). This would allow CC and the client to know that "this sound should be played on all these speakers at the same time." As well, this would mean only one speaker_audio_empty event (perhaps controller_audio_empty?) would need to be queued instead of the current spam of events we get when we have multiple speakers attached.

This is realistic enough as well, we'd not be stepping out of bounds into "meta-ness" or anything with this. My own surround sound system at home only connects the TV to one speaker, the rest are controlled wirelessly by that single speaker.

Pros

• Multiple speaker in-sync playback would be much easier to achieve.
• Ideally speakers would still be able to connect to computers normally, so it wouldn't break backwards compatibility.

Cons

• Wireless in-sync playback would still suffer.
  • I believe if you're trying to play audio over distances where you need a wireless system, you won't be able to hear if it's out of sync with the "main" system anyways.
• Complexity, for both the user and maintainer.
  • New users may think that, in order to use speakers, they must connect the audio cables to the computer rather than the audio controller. This could be mitigated by having the controller just play audio on the same CC network it's connected to and not having a separate cable network, but that just feels slightly weird to me.
  • This would likely require an overhaul of the current audio playback system (or at least, an addon to it), as well as the controller itself.

Footnote

Had this idea while working on some code. I know it is probably a large undertaking to make an entirely new networking system and reworking the audio, but I wanted to write it down in case there were any thoughts about it.

[SquidDev]

Unfortunately I do not believe this is possible with Minecraft's new sound system. Computronics did this by using OpenAL directly, but that's something I'd rather avoid.

[SquidDev]

One neat approach Phonos goes with is to share a sound instance across all connected speakers, and then set the position of that sound to be some weighted average¹ of nearby speakers. This definitely won't work with any of the sound physics mods, but is a pretty clever idea.

What I'm less sure on, is if we want to do audio cables at all. The main advantage of audio cables is really server/network cost - as audio is shared between speakers, you don't need to send multiple identical packets to the client (see also #1313). I don't think implementation is any easier, and it doesn't solve the fundamental issue of audio desync.

I think really the best first step is to characterise the desync people see (how often does this happen, how bad is the desync, single player or multi-player) and then get a better idea of why this happens. Everyone is aware of the desync issue, but I think the only bug report I've received about it is cc-tweaked/cc-restitched#144, which is very different issue!

I can think of a few obvious causes (mostly around audio queueing being split across multiple ticks), but none of that should introduce more than 0.1s delay. My understanding is that sometimes people see much worse desync,

I struggle to reproduce desync locally - both in single player and on a local server, which suggests this is (at least partially!) caused by networking, but I don't know for sure.

Footnotes

1. Mostly for future reference, using this function for weight over distance. ↩

[Lupus590]

Could we detect that the speakers are getting the same audio source and then have the player only able to hear the nearer one? When they move to another speaker playing the same audio the sound should seem to jump to the now nearer speaker.
It'll be odd, but things won't be overlapping and people can still boost the range of sounds via multiple speakers.

Alternatively, I guess that we could make speakers into multi-blocks and the more blocks the more range the sound has, not necessarily louder just travels further.

Neither of these would allow people to do stereo setups but people don't seem to be attempting to do that very often.

[SquidDev]

Could we detect that the speakers are getting the same audio source and then have the player only able to hear the nearer one?

Without knowing what's causing the desync¹, I don't know how reliable such a detection would be.

Footnotes

1. See https://github.com/cc-tweaked/CC-Tweaked/issues/1488#issuecomment-1643676677. Please I really need someone who can reproduce this to do some investigation here! ↩

[MistakingManx]

Is there a mod to increase the stream count for Forge 1.20.1?
There's a base with a lot of levels and each uses a level uses a speaker for announcements, when it plays sounds it stops all pocket computers to stop playing audio period.

[fatboychummy]

I seem to have missed the previous comments here. Oops.

For the desync issue, I have an unfortunate relation with audio in that if I notice certain aspects of it, it'll drive me crazy. I'll admit that the desync isn't a whole lot -- really it's rather minor -- but I can still feel it, and that is what annoys me. I've not had anything worse than the 0.1s figure you mentioned earlier (at least, anything that wasn't caused by my own bad code), but even below that is enough to be well-noticeable. At least, to me.

For characterizing, I could see hashing the input audio buffer and logging the exact time it was queued on the client at as being somewhat helpful. The hash will allow us to see if similar audio buffers have been played at a glance, and the logged time will give an indication of if there was desync, and by how much. I may play around with this (if I can find the time to do so).

For detecting audios having similar streams (via Lupus590's comment), my recommendation would be to move the burden to speaker.playAudio -- add a third argument, id, so the method's signature would now be speaker.playAudio(buffer: string[, volume: number][, id: integer]).

If a speaker attempts to play audio, it first checks if another speaker on the wired network is playing audio with the given ID. If there is one, then it can sync up much more easily and with better performance than by comparing audio streams. That, and we don't get edge-cases where two totally different audio streams "join together", and suddenly some sounds aren't playing.