Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address CA config map creation race condition #19

Merged
merged 2 commits into from
Dec 5, 2023
Merged

Conversation

marc-barry
Copy link
Member

During rapid mutation of pods with the qpoint.io/inject-ca: "true" annotation it is possible that a race condition occurs that leads to the following error:

2023-12-05T05:18:07Z	ERROR	pod.v1.admission.webhook[2e6e597f-f6b9-435d-a7bf-70f608239668]	failed to add assets to namespace for ca injection	{"error": "creating configmap for Qtap CA bundles: configmaps \"qtap-ca-bundle.crt\" already exists"}
github.com/qpoint-io/kubernetes-qtap-operator/api/v1.(*Webhook).Handle
	/workspace/api/v1/webhook.go:94
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/webhook.go:169
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:98
sigs.k8s.io/controller-runtime/pkg/webhook/internal/metrics.InstrumentedHook.InstrumentHandlerInFlight.func1
	/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:60
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2136
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:147
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2136
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:109
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2136
net/http.(*ServeMux).ServeHTTP
	/usr/local/go/src/net/http/server.go:2514
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2938
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:2009

The issue is that the config map already exists due a a race condition with multiple calls to EnsureAssetsInNamespace() and multiple attempts to create the same config map. This pull request locks the write part of the function call so only one can be in-flight at any given time to prevent creation errors.

@marc-barry marc-barry merged commit 31cfed3 into main Dec 5, 2023
1 check passed
@marc-barry marc-barry deleted the marc-barry/ca-race branch December 5, 2023 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants