Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run Minecraft servers in containers through Kubernetes #3

Draft
wants to merge 57 commits into
base: master
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
9ef25b6
Add role to msm-app deployment with access to kubernetes API.
battery-staple Jul 22, 2024
c115802
Add sample monitor deployment
battery-staple Jul 23, 2024
9e861ad
Add jar downloading skeleton to monitor
battery-staple Jul 23, 2024
45ed9ce
Remove extraneous properties and update Java version in monitor Docke…
battery-staple Jul 23, 2024
f28456a
Add KubernetesRunner skeleton and fill out prepareEnvironment
battery-staple Jul 23, 2024
6a84ae3
Improve monitor deployment to use named target ports
battery-staple Jul 23, 2024
8f28e96
Make monitor not download jar each time
battery-staple Jul 23, 2024
18b0f9a
Fix monitor port names to keep them under the 15 character limit
battery-staple Jul 25, 2024
008a7de
Configure traefik gateway to control access to app and running servers
battery-staple Jul 25, 2024
70a4955
Extract service exclusively for minecraft, allowing reassigning that …
battery-staple Jul 25, 2024
d61d313
Allow running servers in Kubernetes
battery-staple Jul 25, 2024
bc42e71
Extract PipingMinecraftServerProcess from MinecraftServerProcessImpl …
battery-staple Jul 26, 2024
29891b2
Implement `MinecraftServerProcess` for Kubernetes servers with `Minec…
battery-staple Jul 26, 2024
26fbb6c
Add SHA-1 hash validation for jar downloads in monitor
battery-staple Jul 27, 2024
e5ba219
Removed double Base64 encoding of monitor deployment secret
battery-staple Jul 27, 2024
e2b77c3
Fixed websocket connection between app and monitor
battery-staple Jul 27, 2024
876c3e8
Fixed creation and running of Minecraft server pods
battery-staple Jul 27, 2024
22139d9
Allow usage of the Kubernetes runner to the frontend
battery-staple Jul 27, 2024
b893ef4
Change monitor names to include their server's UUID
battery-staple Jul 27, 2024
44bc9b4
Use random tokens for communication with monitors
battery-staple Jul 27, 2024
96f2eff
Inject Kubernetes API Client and associated objects
battery-staple Jul 27, 2024
db0c813
Persist Kubernetes environments between runs
battery-staple Jul 27, 2024
9b19663
Persist and use random auth tokens
battery-staple Jul 29, 2024
7ec70de
Fix monitor PVC name
battery-staple Jul 29, 2024
4c0961a
Fix bug where some console messages were not sent to the frontend
battery-staple Jul 29, 2024
e92f405
Fix initialization order bug preventing cleanup of old runs
battery-staple Jul 29, 2024
23a3ba6
Refactor PipingMinecraftServerProcess to be more idiomatic Kotlin
battery-staple Jul 29, 2024
d626d34
Recreate Websocket connection to monitor pods after they expire
battery-staple Jul 31, 2024
4a1bfe8
Add error handling and exponential backoff to make monitor Websocket …
battery-staple Jul 31, 2024
cad26b6
Remove repeated logging of each Websocket message. This declutters th…
battery-staple Jul 31, 2024
dfed39a
Refactor websocket connection logic in MinecraftServerPod
battery-staple Jul 31, 2024
b30b946
Introduce tryWithBackoff to simplify backoff in websocket and databas…
battery-staple Aug 1, 2024
87bb17f
Allow MinecraftServerPod to watch the kubernetes pod and update when …
battery-staple Aug 1, 2024
4611a42
Fix persistence of `LocalMinecraftServerEnvironment`s
battery-staple Aug 1, 2024
b92fc03
Refactor MinecraftServerPod to improve reliability, including recover…
battery-staple Aug 17, 2024
77c337f
Rename MinecraftServerPod to DeploymentProcess. Since this class now …
battery-staple Aug 17, 2024
a2b1208
Prevent KubernetesRunner#prepareEnvironment from failing silently
battery-staple Aug 17, 2024
c479404
Implement KubernetesRunner#cleanupEnvironment
battery-staple Aug 18, 2024
58afef9
Move PipingMinecraftServerProcess to its own file
battery-staple Aug 18, 2024
a799325
Fix a bug where a CurrentRun that continues while the application fai…
battery-staple Aug 19, 2024
7ca6da9
Ensure `MinecraftServerRunner`s only clean up their own `CurrentRunRe…
battery-staple Aug 20, 2024
711df0d
Clean up monitor code
battery-staple Aug 20, 2024
4f57d6e
Extract request building logic in monitor into function
battery-staple Aug 20, 2024
110ecad
Fix accidental usage of the Watch logger in other files
battery-staple Aug 20, 2024
9a01df5
Add necessary permissions to allow deleting Kubernetes servers
battery-staple Aug 22, 2024
de62b0c
Handle deletion of K8s servers when some components have been (errone…
battery-staple Aug 23, 2024
316fb8f
Improve logging in `RestAPIServiceImple#createServer`
battery-staple Jan 5, 2025
e3e3986
Make `KubernetesEnvironment` take a `MinecraftServer` rather than jus…
battery-staple Jan 5, 2025
99ab309
Stop running Kubernetes servers immediately when created (wait until …
battery-staple Jan 5, 2025
0630c3b
Save log files by run UUID on monitor for later access
battery-staple Jan 7, 2025
af9ac23
Make MonitorTokenRepository#getTokenForServer correctly return null w…
battery-staple Jan 8, 2025
71cd545
Improve documentation and logging in `KubernetesRunner#initialProcess`
battery-staple Jan 10, 2025
c96f59f
Fix log message order in the websocket console route
battery-staple Jan 10, 2025
3598d5d
Move `MutexGuardedResource` to the shared library
battery-staple Jan 10, 2025
3e44950
Add more idiomatic factory methods for initializing `MutexGuardedReso…
battery-staple Jan 10, 2025
8c3def7
Move custom status page exceptions into shared library
battery-staple Jan 12, 2025
ed7eb77
Add optional messages to status page exceptions
battery-staple Jan 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix bug where some console messages were not sent to the frontend
battery-staple committed Jul 29, 2024
commit 4c0961a8db26e4d072fa7cf0439b8c0772849d44
Original file line number Diff line number Diff line change
@@ -70,7 +70,7 @@ interface MinecraftServerProcess {
}
}

abstract class PipingMinecraftServerProcess(private val serverName: String) : MinecraftServerProcess {
abstract class PipingMinecraftServerProcess(protected val serverName: String) : MinecraftServerProcess {
override val output: Flow<ProcessMessage<Output>> by lazy {
assertInv()
_output.asSharedFlow()
Original file line number Diff line number Diff line change
@@ -27,6 +27,7 @@ import io.ktor.client.*
import io.ktor.client.engine.okhttp.*
import io.ktor.client.plugins.contentnegotiation.*
import io.ktor.client.plugins.websocket.*
import io.ktor.serialization.kotlinx.*
import io.ktor.serialization.kotlinx.json.*
import io.ktor.server.application.*
import io.ktor.server.engine.*
@@ -71,7 +72,9 @@ fun Application.module() {
json(Json { ignoreUnknownKeys = true })
}

install(WebSockets)
install(WebSockets) {
contentConverter = KotlinxWebsocketSerializationConverter(Json { ignoreUnknownKeys = false })
}
}
}
single<Json> { Json { ignoreUnknownKeys = false } }
Original file line number Diff line number Diff line change
@@ -38,24 +38,23 @@ class MinecraftServerPod(
session.sendSerialized(ConsoleMessageAPIModel.Input(input))
}

override suspend fun getStdOut(): Flow<String> {
val incoming = incomingMessages.await()
return incoming
.filterIsInstance<ConsoleMessageAPIModel.Output.Log>()
.map { message -> message.text }
}
/**
* Backing field for [getStdOut]
*/
private val stdout = MutableSharedFlow<String>()
override suspend fun getStdOut(): Flow<String> = stdout.asSharedFlow()

override suspend fun getStdErr(): Flow<String> {
val incoming = incomingMessages.await()
return incoming
.filterIsInstance<ConsoleMessageAPIModel.Output.ProcessError>()
.map { message -> message.text }
}
/**
* Backing field for [getStdErr]
*/
private val stderr = MutableSharedFlow<String>()
override suspend fun getStdErr(): Flow<String> = stderr.asSharedFlow()

override suspend fun waitForExit(): Int? {
incomingMessages.await().collect()
return null
}
/**
* The code with which the pod's process exited
*/
private val exitCode = CompletableDeferred<Int?>()
override suspend fun waitForExit(): Int? = exitCode.await()

override suspend fun stop(softTimeout: Duration, additionalForcibleTimeout: Duration): Int? {
TODO()
@@ -77,27 +76,52 @@ class MinecraftServerPod(
}
}

private val json: Json by inject()
private val logger = LoggerFactory.getLogger(MinecraftServerPod::class.java)

/**
* The parsed messages sent from the pod.
* Handles the incoming frames from the websocket
*/
private val incomingMessages: Deferred<Flow<ConsoleMessageAPIModel.Output>> = coroutineScope.async {
val session = session.await()

val messages = session.incoming
private suspend fun DefaultClientWebSocketSession.handleIncoming() {
val messages = incoming
.receiveAsFlow()
.onEach { if (it !is Frame.Text) logger.warn("Received non-text frame {}", it) }
.onEach { if (it !is Frame.Text) logger.warn("Received non-text frame {}", it) else logger.trace("Incoming message {} from server {}", it.readText(), serverName) }
.filterIsInstance<Frame.Text>()
.map { it.readText() }
.map { json.decodeFromString<ConsoleMessageAPIModel.Output>(it)}
.onCompletion { logger.debug("Incoming messages from pod for server {} ended", serverName) }

messages
messages.collect { message ->
val outputFlow = when (message) {
is ConsoleMessageAPIModel.Output.Log -> stdout
is ConsoleMessageAPIModel.Output.ProcessError -> stderr
}

outputFlow.emit(message.text)
}
}

private val json: Json by inject()
private val logger = LoggerFactory.getLogger(MinecraftServerPod::class.java)
/**
* Handles the closing of the websocket
*/
private suspend fun DefaultClientWebSocketSession.handleEnd() {
closeReason.await()
// if we've reached here, collection has ended which means the websocket has closed. TODO: better way to detect ending
exitCode.complete(null) // TODO: actual exit code
}

init {
initIO()

coroutineScope.launch {
val session = session.await()
launch {
session.handleIncoming()
}

launch {
session.handleEnd()
}
}
}
}