testing in air-gapped: TLS handshake error

[NohaIhab]

Bug Description

when deploying admission-webhook charm in an air-gapped environment, the charm goes to active, but the workload container has logs of TLS handshake errors. This indicates that the certificates configured in the MutatingWebhookConfiguration cannot be verified, and the charm will not act as expected in the bundle.

To Reproduce

After setting up the air-gapped environment, go the directory where the charm file is, and deploy the charm with --resource oci-image set to the image in the local registry.
juju deploy ./admission-webhook_98aac65.charm --resource oci-image=172.17.0.2:5000/kubeflownotebookswg/poddefaults-webhook:v1.7.0 --trust

Environment

following the script in canonical/bundle-kubeflow#682 (comment)

Relevant Log Output

microk8s kubectl logs admission-webhook-0 -c admission-webhook -nkubeflow
2023-08-29T14:09:04.466Z [pebble] HTTP API server listening on ":38813".
2023-08-29T14:09:04.466Z [pebble] Started daemon.
2023-08-29T14:09:15.820Z [pebble] POST /v1/files 5.123433ms 200
2023-08-29T14:09:15.824Z [pebble] POST /v1/files 3.933732ms 200
2023-08-29T14:09:16.800Z [pebble] GET /v1/plan?format=yaml 302.664µs 200
2023-08-29T14:09:16.802Z [pebble] POST /v1/layers 415.125µs 200
2023-08-29T14:09:16.811Z [pebble] POST /v1/services 4.067584ms 202
2023-08-29T14:09:16.815Z [pebble] Service "admission-webhook" starting: /webhook
2023-08-29T14:09:16.832Z [admission-webhook] I0829 14:09:16.832366      13 main.go:768] About to start serving webhooks: &http.Server{Addr:":4443", Handler:http.Handler(nil), TLSConfig:(*tls.Config)(0xc000496480), ReadTimeout:0, ReadHeaderTimeout:0, WriteTimeout:0, IdleTimeout:0, MaxHeaderBytes:0, TLSNextProto:map[string]func(*http.Server, *tls.Conn, http.Handler)(nil), ConnState:(func(net.Conn, http.ConnState))(nil), ErrorLog:(*log.Logger)(nil), BaseContext:(func(net.Listener) context.Context)(nil), ConnContext:(func(context.Context, net.Conn) context.Context)(nil), inShutdown:0, disableKeepAlives:0, nextProtoOnce:sync.Once{done:0x0, m:sync.Mutex{state:0, sema:0x0}}, nextProtoErr:error(nil), mu:sync.Mutex{state:0, sema:0x0}, listeners:map[*net.Listener]struct {}(nil), activeConn:map[*http.conn]struct {}(nil), doneChan:(chan struct {})(nil), onShutdown:[]func()(nil)}
2023-08-29T14:09:17.825Z [pebble] GET /v1/changes/1/wait?timeout=4.000s 1.01415687s 200
2023-08-29T14:09:19.456Z [pebble] GET /v1/plan?format=yaml 367.09µs 200
2023-08-29T14:09:46.804Z [admission-webhook] 2023/08/29 14:09:46 http: TLS handshake error from [::1]:36654: EOF
2023-08-29T14:10:16.803Z [admission-webhook] 2023/08/29 14:10:16 http: TLS handshake error from [::1]:37596: EOF
2023-08-29T14:10:46.803Z [admission-webhook] 2023/08/29 14:10:46 http: TLS handshake error from [::1]:41522: EOF
2023-08-29T14:11:00.597Z [pebble] GET /v1/plan?format=yaml 189.604µs 200
2023-08-29T14:11:05.386Z [pebble] GET /v1/checks 107.768µs 200
2023-08-29T14:11:16.803Z [admission-webhook] 2023/08/29 14:11:16 http: TLS handshake error from [::1]:44022: EOF
2023-08-29T14:11:35.521Z [pebble] GET /v1/plan?format=yaml 166.75µs 200
2023-08-29T14:11:41.220Z [pebble] GET /v1/checks 43.539µs 200
2023-08-29T14:11:46.804Z [admission-webhook] 2023/08/29 14:11:46 http: TLS handshake error from [::1]:33436: EOF
2023-08-29T14:12:16.803Z [admission-webhook] 2023/08/29 14:12:16 http: TLS handshake error from [::1]:46958: EOF
2023-08-29T14:12:46.803Z [admission-webhook] 2023/08/29 14:12:46 http: TLS handshake error from [::1]:34576: EOF
2023-08-29T14:13:16.804Z [admission-webhook] 2023/08/29 14:13:16 http: TLS handshake error from [::1]:41790: EOF
2023-08-29T14:13:46.802Z [admission-webhook] 2023/08/29 14:13:46 http: TLS handshake error from [::1]:53510: EOF
2023-08-29T14:13:51.783Z [pebble] GET /v1/checks?names=admission-webhook-up 52.829µs 200
2023-08-29T14:14:16.803Z [admission-webhook] 2023/08/29 14:14:16 http: TLS handshake error from [::1]:42772: EOF
2023-08-29T14:14:46.803Z [admission-webhook] 2023/08/29 14:14:46 http: TLS handshake error from [::1]:55118: EOF
2023-08-29T14:15:16.802Z [admission-webhook] 2023/08/29 14:15:16 http: TLS handshake error from [::1]:49968: EOF
2023-08-29T14:15:46.803Z [admission-webhook] 2023/08/29 14:15:46 http: TLS handshake error from [::1]:42028: EOF
2023-08-29T14:16:16.802Z [admission-webhook] 2023/08/29 14:16:16 http: TLS handshake error from [::1]:54382: EOF
2023-08-29T14:16:46.803Z [admission-webhook] 2023/08/29 14:16:46 http: TLS handshake error from [::1]:60010: EOF
2023-08-29T14:17:16.803Z [admission-webhook] 2023/08/29 14:17:16 http: TLS handshake error from [::1]:52832: EOF
2023-08-29T14:17:46.803Z [admission-webhook] 2023/08/29 14:17:46 http: TLS handshake error from [::1]:60104: EOF
2023-08-29T14:18:16.803Z [admission-webhook] 2023/08/29 14:18:16 http: TLS handshake error from [::1]:37882: EOF
2023-08-29T14:18:46.803Z [admission-webhook] 2023/08/29 14:18:46 http: TLS handshake error from [::1]:36522: EOF
2023-08-29T14:18:47.005Z [pebble] GET /v1/checks?names=admission-webhook-up 47.519µs 200
2023-08-29T14:19:16.803Z [admission-webhook] 2023/08/29 14:19:16 http: TLS handshake error from [::1]:39050: EOF

[NohaIhab]

tested with the simple poddefault example in /tests/integration in an air-gapped environment, the poddefault was applied as expected in spite of the TLS errors.
workload container logs:

2023-09-01T07:55:51.005Z [pebble] GET /v1/checks?names=admission-webhook-up 52.665µs 200
2023-09-01T07:56:15.821Z [admission-webhook] 2023/09/01 07:56:15 http: TLS handshake error from [::1]:42064: EOF
2023-09-01T07:56:45.821Z [admission-webhook] 2023/09/01 07:56:45 http: TLS handshake error from [::1]:46242: EOF
2023-09-01T07:57:15.822Z [admission-webhook] 2023/09/01 07:57:15 http: TLS handshake error from [::1]:36116: EOF
2023-09-01T07:57:29.319Z [admission-webhook] I0901 07:57:29.319109      13 main.go:598] Entering mutatePods in mutating webhook
2023-09-01T07:57:29.319Z [admission-webhook] I0901 07:57:29.319360      13 main.go:624] Looking at pod annotations, found: map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"access-ml-pipeline":"true"},"name":"testpod","namespace":"test-admission-webhook-user-namespace"},"spec":{"containers":[{"args":["while true; do sleep 3600; done"],"command":["/bin/bash","-c","--"],"image":"172.17.0.2:5000/minio/minio:RELEASE.2021-09-03T03-56-13Z","imagePullPolicy":"Always","name":"ubuntu"}]}}
2023-09-01T07:57:29.319Z [admission-webhook] ]
2023-09-01T07:57:29.524Z [admission-webhook] I0901 07:57:29.523992      13 main.go:644] fetched 1 poddefault(s) in namespace test-admission-webhook-user-namespace
2023-09-01T07:57:29.524Z [admission-webhook] I0901 07:57:29.524048      13 main.go:660] 1 matching pod defaults, for pod testpod
2023-09-01T07:57:29.524Z [admission-webhook] I0901 07:57:29.524059      13 main.go:666] Matching PD detected of count 1, patching spec
2023-09-01T07:57:29.524Z [admission-webhook] I0901 07:57:29.524083      13 main.go:479] mutating pod: testpod
2023-09-01T07:57:29.524Z [admission-webhook] I0901 07:57:29.524102      13 main.go:681] applied poddefaults: access-ml-pipeline successfully on Pod: testpod 
2023-09-01T07:57:45.821Z [admission-webhook] 2023/09/01 07:57:45 http: TLS handshake error from [::1]:41702: EOF
2023-09-01T07:58:15.821Z [admission-webhook] 2023/09/01 07:58:15 http: TLS handshake error from [::1]:51300: EOF
2023-09-01T07:58:45.821Z [admission-webhook] 2023/09/01 07:58:45 http: TLS handshake error from [::1]:52200: EOF

It seems that the TLS error does not affect the functionality of the webhook operator.
will need to confirm this when doing bundle testing in airgapped.

[NohaIhab]

tried creating a notebook in air-gapped, the pod-default wasn't applied as expected
logs from admission-webhook:

2023-09-21T22:55:45.570Z [admission-webhook] 2023/09/21 22:55:45 http: TLS handshake error from [::1]:56262: EOF
2023-09-21T22:56:15.570Z [admission-webhook] 2023/09/21 22:56:15 http: TLS handshake error from [::1]:41634: EOF
2023-09-21T22:56:45.570Z [admission-webhook] 2023/09/21 22:56:45 http: TLS handshake error from [::1]:53972: EOF
2023-09-21T22:56:58.329Z [pebble] GET /v1/checks?names=admission-webhook-up 54.67µs 200
2023-09-21T22:57:15.570Z [admission-webhook] 2023/09/21 22:57:15 http: TLS handshake error from [::1]:55470: EOF
2023-09-21T22:57:38.939Z [admission-webhook] I0921 22:57:38.939819      13 main.go:598] Entering mutatePods in mutating webhook
2023-09-21T22:57:38.940Z [admission-webhook] I0921 22:57:38.940462      13 main.go:598] Entering mutatePods in mutating webhook
2023-09-21T22:57:40.007Z [admission-webhook] I0921 22:57:40.007746      13 request.go:665] Waited for 1.01705739s due to client-side throttling, not priority and fairness, request: GET:https://10.152.183.1:443/apis/machinelearning.seldon.io/v1?timeout=32s
2023-09-21T22:57:40.615Z [admission-webhook] I0921 22:57:40.614642      13 main.go:644] fetched 1 poddefault(s) in namespace profilename
2023-09-21T22:57:40.617Z [admission-webhook] I0921 22:57:40.617480      13 main.go:644] fetched 1 poddefault(s) in namespace profilename
2023-09-21T22:57:45.570Z [admission-webhook] 2023/09/21 22:57:45 http: TLS handshake error from [::1]:58420: EOF
2023-09-21T22:58:15.570Z [admission-webhook] 2023/09/21 22:58:15 http: TLS handshake error from [::1]:44606: EOF
2023-09-21T22:58:16.417Z [admission-webhook] I0921 22:58:16.416987      13 main.go:598] Entering mutatePods in mutating webhook
2023-09-21T22:58:17.467Z [admission-webhook] I0921 22:58:17.467713      13 request.go:665] Waited for 1.046999552s due to client-side throttling, not priority and fairness, request: GET:https://10.152.183.1:443/apis/apiextensions.k8s.io/v1?timeout=32s
2023-09-21T22:58:18.073Z [admission-webhook] I0921 22:58:18.073845      13 main.go:644] fetched 1 poddefault(s) in namespace profilename
2023-09-21T22:58:45.570Z [admission-webhook] 2023/09/21 22:58:45 http: TLS handshake error from [::1]:52854: EOF
2023-09-21T22:59:15.569Z [admission-webhook] 2023/09/21 22:59:15 http: TLS handshake error from [::1]:53128: EOF
2023-09-21T22:59:45.570Z [admission-webhook] 2023/09/21 22:59:45 http: TLS handshake error from [::1]:47726: EOF
2023-09-21T23:00:15.570Z [admission-webhook] 2023/09/21 23:00:15 http: TLS handshake error from [::1]:45644: EOF

I still need to investigate what is causing this, whether it's related to the TLS handshake error.

[NohaIhab]

This issue is possibly related to kubeflow/kubeflow#6708
We're not sure why it's only happening in air-gapped