From 1f8b2a312aacfb589204233b8d48c9367cb99fe1 Mon Sep 17 00:00:00 2001 From: Dimitar Dimitrov Date: Tue, 10 Dec 2024 20:12:52 +0100 Subject: [PATCH] Set timeout for k8s HTTP client (#186) * Set timeout for k8s HTTP client We had a request to an ingesters take 20m and block the reconiliation loop. We want to avoid that, so adding a timeout. We use this client for both k8s and StatefulSet requests. __I assume the operations that the StatefulSet requests will always be fast and will never have to actually do a lot of work (like uploading blocks). Can the reviewers confirm this assumption?__ * Add CHANGELOG.md entry PR number --- CHANGELOG.md | 3 ++- cmd/rollout-operator/main.go | 4 ++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 54afb43a..fe009c49 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,8 @@ ## main / unreleased * [ENHANCEMENT] New parameter log.format allows to set logging format to logfmt (default) or json (new). #184 - +* [ENHANCEMENT] Add a 5 minute timeout to requests to Pods and to the Kubernetes control plane. #186 + ## v0.21.0 * [ENHANCEMENT] Log debug information about StatefulSets as they are created, updated and deleted. #182 diff --git a/cmd/rollout-operator/main.go b/cmd/rollout-operator/main.go index 0d1d6d1a..8b3a8e39 100644 --- a/cmd/rollout-operator/main.go +++ b/cmd/rollout-operator/main.go @@ -276,6 +276,10 @@ func buildKubeConfig(apiURL, cfgFile string) (*rest.Config, error) { if err != nil { return nil, err } + // Set a generous timeout. + // We use this client for various HTTP operations against the k8s API and against the StatefulSets. + // We want to not be stuck waiting forever on a TCP timeout, but also not interrupt any process the StatefulSets might eb doing. + config.Timeout = 5 * time.Minute return config, nil }