High ping and choppy audio

[sheitmaster]

I'm experiencing issues with the bot.
I've got it hosted in a docker on my unraid-server, it's on a 500Mbit fiber connection and i'm seeing 230ms ping and 120ms websocket ping.
The audio is stuttering and sounds really bad.

There's nothing i can see that may be causing this, the system is more than capable of running this afaik, unless homeassistant controlling 9 bulbs and 3 remotes and a hue bridge is enough to interfere.

Anyone have any good ideas as to what might be causing it?

I'm also experiencing packet loss when i speak in discord, but only when the bot is connected or has been connected.
If i then stop the music and the bot leaves, i still have packet loss, but if i also leave the channel and come back, packetloss is gone.