feat(api): Replace fluentd deployment with statefulset

[deadlycoconuts]

Context

In order to make the Fluentd deployment used by a Turing router deployment (to flush logs to BigQuery) more robust to cluster autoscaling events, which may remove the node on which the one and only Fluentd pod is scheduled and cause the Turing routers to return errors, we need to increase the number of replicas of the Fluentd deployment from 1 to 2.

However, doing so involves replacing how Turing deploys the Fluentd deployment - from a Kubernetes Deployment resource to a StatefulSet. This is needed because deployments do not allow its pod replicas to specify volume/volume mount names that are different. This is crucial because the PVC that the current Fluentd pod uses is of the ReadWriteOnce type, which means that Fluentd replicas need to be scheduled on the same node in order to read/write on the same PVC. Other access mode exist - such as ReadOnlyMany and ReadWriteMany. However, ReadOnlyMany wouldn't allow Fluentd to write anything to the PVC, completely voiding the original purpose of even attaching a PVC to Fluentd, while ReadWriteMany isn't available in all managed Kubernetes services.

By replacing the Deployment with a StatefulSet, we are able to specify a generic name for any of the PVCs that will be spun up in tandem with the number of pods the StatefulSet controls. This allows us to increase the number of Fluentd pods to 2, which allows for better uptime of the Fluentd service, especially when we introduce other configurations such as PodDisruptionBudgets.

As we are increasing the number of Fluentd pods, we are also halving the amount of CPU and memory resources requested for each pod to keep the total amount of resources consumed the same as before.

Additional Changes

This PR also includes a tiny change to the pod disruption budget selector key used for Knative services (Turing routers, enrichers and ensemblers) from app to serving.knative.dev/service. This is done because the label value corresponding to the app key on these Knative service-managed pods have an additional suffix (e.g. -0) that indicates the revision number of the deployment.

While this isn't a problem in most instances, any manual edits made to the Knative service manifest would undoubtedly cause the revision number to be bumped (e.g. from 0 to 1), making the existing pod disruption budget resources created useless. This isn't the case however, for the label value corresponding to the serving.knative.dev/service key, which does not have the additional revision number suffix.

Main Modifications

• api/turing/cluster/controller.go - Refactoring of the steps to deploy Fluentd as a stateful set instead of a deployment
• api/turing/cluster/kubernetes_service.go - Refactoring of the Fluentd deployment spec to a statefulset spec
• api/turing/cluster/servicebuilder/fluentd.go - Halving of Fluentd CPU and memory resources for each pod
• api/turing/cluster/servicebuilder/service_builder.go - Addition of steps to configure PDBs differently for Fluentd
• api/turing/service/router_deployment_service.go - Removal of steps needed to manually create/delete PVCs

[leonlnj]

thanks, lgtm!

[deadlycoconuts]

Thanks for the reviews everyone! Merging this now! 🚀