[BUG] Secrets injection is broken

[eapolinario]

Describe the bug

flyte pod webhook panics while injecting secrets, regardless of the secret manager (k8s, aws, gcp, etc).

We see this panic in the logs when executing a task that requires a secret:

{"json":{"exec_id":"f3d082a54f9b94988ba4","node":"workflowsexamplesecrettask","ns":"flytesnacks-development","res_ver":"108245851","routine":"worker-3","src":"controller.go:36","tasktype":"python-task","wf":"flytesnacks:development:.flytegen.workflows.example.secret_task"},"level":"info","msg":"The back-off handler for [/v1, Kind=pod,flytesnacks-development] has been loaded.\n","ts":"2023-11-30T21:26:53Z"}
2023/11/30 21:26:53 http: panic serving 10.2.149.237:48966: runtime error: invalid memory address or nil pointer dereference
goroutine 26608 [running]:
net/http.(*conn).serve.func1()
        /usr/local/go/src/net/http/server.go:1850 +0xbf
panic({0x2f4dc00, 0x6164a90})
        /usr/local/go/src/runtime/panic.go:890 +0x262
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Decoder).DecodeRaw(0x0, {{0xc005ce8000, 0x1b70, 0x2000}, {0x0, 0x0}}, {0x46ca258, 0xc0037b0000})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/decode.go:77 +0x15a
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Decoder).Decode(_, {{{0xc00377fcb0, 0x24}, {{0x0, 0x0}, {0xc005fd30c0, 0x2}, {0xc005fd30c2, 0x3}}, {{0x0, ...}, ...}, ...}}, ...)
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/decode.go:49 +0x8e
github.com/flyteorg/flyte/flytepropeller/pkg/webhook.(*PodMutator).Handle(_, {_, _}, {{{0xc00377fcb0, 0x24}, {{0x0, 0x0}, {0xc005fd30c0, 0x2}, {0xc005fd30c2, ...}}, ...}})
        /flyteorg/build/flytepropeller/pkg/webhook/pod.go:86 +0xc8
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(_, {_, _}, {{{0xc00377fcb0, 0x24}, {{0x0, 0x0}, {0xc005fd30c0, 0x2}, {0xc005fd30c2, ...}}, ...}})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/webhook.go:169 +0x20b
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc008872a50, {0x7f1ba49b1e40?, 0xc00a05f450}, 0xc005cc0100)
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:98 +0xc94
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7f1ba49b1e40, 0xc00a05f450}, 0x46e7500?)
        /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:60 +0xd4
net/http.HandlerFunc.ServeHTTP(0x46e7560?, {0x7f1ba49b1e40?, 0xc00a05f450?}, 0x7e1286?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x46e7560?, 0xc00175bce0?}, 0xc005cc0100)
        /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:147 +0xc5
net/http.HandlerFunc.ServeHTTP(0x7e0ce5?, {0x46e7560?, 0xc00175bce0?}, 0x40?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2({0x46e7560, 0xc00175bce0}, 0xc005cc0100)
        /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:109 +0xc7
net/http.HandlerFunc.ServeHTTP(0xc00175bce0?, {0x46e7560?, 0xc00175bce0?}, 0x34ec45f?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc00377fc35?, {0x46e7560, 0xc00175bce0}, 0xc005cc0100)
        /usr/local/go/src/net/http/server.go:2487 +0x149
net/http.serverHandler.ServeHTTP({0x46cec18?}, {0x46e7560, 0xc00175bce0}, 0xc005cc0100)
        /usr/local/go/src/net/http/server.go:2947 +0x30c
net/http.(*conn).serve(0xc000c6a8c0, {0x46e9780, 0xc008ab12f0})
        /usr/local/go/src/net/http/server.go:1991 +0x607
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:3102 +0x4db

We recently bumped the version of sigs.k8s.io/controller-runtime which included a breaking change made in https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.15.0.

Expected behavior

Secret injection works.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[dosubot]

🤖

Hello @eapolinario!

I'm Dosu, a bot here to assist in addressing issues, answering questions, and guiding you to become a contributor while we're waiting for a human maintainer. Don't worry, we're already taking a look at your issue with the secrets injection and will provide a complete response shortly.

[EngHabu]

Fixed