Integrate `PodDefaults` rock #153

mvlassis · 2024-09-02T09:20:05Z

Closes #151

This PR integrates the PodDefaults rock into admission-webhook for the CKF 1.9 bundle.

The new OCI image has been published in Dockerhub here.

mvlassis · 2024-09-13T16:18:16Z

Ok, a simple change of the rock doesn't seem to work, the charm stays on blocked with the following logs from the admission-webhook container:

juju debug-log -i admission-webhook --replay 
unit-admission-webhook-0: 15:49:58 INFO juju.cmd running containerAgent [3.4.5 e5130b8c9e327c81818c849951575b9f3156307d gc go1.21.12]
unit-admission-webhook-0: 15:49:58 INFO juju.cmd.containeragent.unit start "unit"
unit-admission-webhook-0: 15:49:58 INFO juju.worker.upgradesteps upgrade steps for 3.4.5 have already been run.
unit-admission-webhook-0: 15:49:58 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301
unit-admission-webhook-0: 15:49:58 INFO juju.api cannot resolve "controller-service.controller-uk8sx.svc.cluster.local": lookup controller-service.controller-uk8sx.svc.cluster.local: operation was canceled
unit-admission-webhook-0: 15:49:58 INFO juju.api connection established to "wss://10.152.183.154:17070/model/6e8f958a-55c4-4cb3-81d6-e46a37fa48fa/api"
unit-admission-webhook-0: 15:49:58 INFO juju.worker.apicaller [6e8f95] "unit-admission-webhook-0" successfully connected to "10.152.183.154:17070"
unit-admission-webhook-0: 15:49:58 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-admission-webhook-0: 15:49:58 INFO juju.worker.logger logger worker started
unit-admission-webhook-0: 15:49:58 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-admission-webhook-0: 15:49:58 INFO juju.worker.leadership admission-webhook/0 promoted to leadership of admission-webhook
unit-admission-webhook-0: 15:49:58 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-admission-webhook-0: 15:49:58 INFO juju.worker.caasupgrader unblocking abort check
unit-admission-webhook-0: 15:49:58 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-admission-webhook-0
unit-admission-webhook-0: 15:49:58 INFO juju.worker.uniter unit "admission-webhook/0" started
unit-admission-webhook-0: 15:49:58 INFO juju.worker.uniter resuming charm install
unit-admission-webhook-0: 15:49:58 INFO juju.worker.uniter.charm downloading local:admission-webhook-0 from API server
unit-admission-webhook-0: 15:49:58 INFO juju.downloader downloading from local:admission-webhook-0
unit-admission-webhook-0: 15:49:58 INFO juju.downloader download complete ("local:admission-webhook-0")
unit-admission-webhook-0: 15:49:58 INFO juju.downloader download verified ("local:admission-webhook-0")
unit-admission-webhook-0: 15:50:06 INFO juju.worker.uniter hooks are retried true
unit-admission-webhook-0: 15:50:06 INFO juju.worker.uniter found queued "install" hook
unit-admission-webhook-0: 15:50:07 INFO unit.admission-webhook/0.juju-log Running legacy hooks/install.
unit-admission-webhook-0: 15:50:08 INFO unit.admission-webhook/0.juju-log Generating certificates if missing.
unit-admission-webhook-0: 15:50:08 INFO unit.admission-webhook/0.juju-log Generating certificates..
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install Generating RSA private key, 2048 bit long modulus (2 primes)
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install ....+++++
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install .........................................+++++
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install e is 65537 (0x010001)
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install Generating RSA private key, 2048 bit long modulus (2 primes)
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install ..............................................................................................+++++
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install ...............................................................................................................+++++
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install e is 65537 (0x010001)
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install Signature ok
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install Getting CA Private Key
unit-admission-webhook-0: 15:50:08 INFO unit.admission-webhook/0.juju-log Certificates already exist.
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: GET https://10.152.183.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log Rendering manifests
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-admin?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-edit?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-view?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log Reconcile completed successfully
unit-admission-webhook-0: 15:50:10 INFO unit.admission-webhook/0.juju-log HTTP Request: GET https://10.152.183.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:10 INFO unit.admission-webhook/0.juju-log Rendering manifests
unit-admission-webhook-0: 15:50:10 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/poddefaults.kubeflow.org?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:10 INFO unit.admission-webhook/0.juju-log Reconcile completed successfully
unit-admission-webhook-0: 15:50:10 INFO unit.admission-webhook/0.juju-log Pebble plan updated with new configuration, replanning
unit-admission-webhook-0: 15:50:11 ERROR unit.admission-webhook/0.juju-log Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/charmed_kubeflow_chisme/pebble/_update_layer.py", line 31, in update_layer
    container.replan()
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/model.py", line 2252, in replan
    self._pebble.replan_services()
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/pebble.py", line 1993, in replan_services
    return self._services_action('replan', [], timeout, delay)
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/pebble.py", line 2090, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "base-admission-webhook" (cannot start service: exited quickly with code 0)
----- Logs from task 1 -----
2024-09-13T15:50:11Z INFO Most recent service output:
    {"level":"info","ts":"2024-09-13T15:50:11Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
    I0913 15:50:11.802163      21 main.go:771] About to start serving webhooks: &http.Server{Addr:":4443", Handler:http.Handler(nil), DisableGeneralOptionsHandler:false, TLSConfig:(*tls.Config)(0xc0001f0d00), ReadTimeout:0, ReadHeaderTimeout:0, WriteTimeout:0, IdleTimeout:0, MaxHeaderBytes:0, TLSNextProto:map[string]func(*http.Server, *tls.Conn, http.Handler)(nil), ConnState:(func(net.Conn, http.ConnState))(nil), ErrorLog:(*log.Logger)(nil), BaseContext:(func(net.Listener) context.Context)(nil), ConnContext:(func(context.Context, net.Conn) context.Context)(nil), inShutdown:atomic.Bool{_:atomic.noCopy{}, v:0x0}, disableKeepAlives:atomic.Bool{_:atomic.noCopy{}, v:0x0}, nextProtoOnce:sync.Once{done:0x0, m:sync.Mutex{state:0, sema:0x0}}, nextProtoErr:error(nil), mu:sync.Mutex{state:0, sema:0x0}, listeners:map[*net.Listener]struct {}(nil), activeConn:map[*http.conn]struct {}(nil), onShutdown:[]func()(nil), listenerGroup:sync.WaitGroup{noCopy:sync.noCopy{}, state:atomic.Uint64{_:atomic.noCopy{}, _:atomic.align64{}, v:0x0}, sema:0x0}}
    {"level":"info","ts":"2024-09-13T15:50:11Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
2024-09-13T15:50:11Z ERROR cannot start service: exited quickly with code 0
-----

unit-admission-webhook-0: 15:50:11 ERROR unit.admission-webhook/0.juju-log Failed to handle <InstallEvent via AdmissionWebhookCharm/on/install[1]> with error: Failed to replan
unit-admission-webhook-0: 15:50:12 INFO unit.admission-webhook/0.juju-log HTTP Request: GET https://10.152.183.1/api/v1/namespaces/kubeflow/services/admission-webhook "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:12 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/kubeflow/services/admission-webhook "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:12 INFO unit.admission-webhook/0.juju-log Kubernetes service 'admission-webhook' patched successfully
unit-admission-webhook-0: 15:50:12 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-admission-webhook-0: 15:50:13 INFO unit.admission-webhook/0.juju-log logging:1: Generating certificates if missing.
unit-admission-webhook-0: 15:50:13 INFO unit.admission-webhook/0.juju-log logging:1: Certificates already exist.
unit-admission-webhook-0: 15:50:13 INFO juju.worker.uniter.operation ran "logging-relation-created" hook (via hook dispatching script: dispatch)
unit-admission-webhook-0: 15:50:13 INFO juju.worker.uniter found queued "leader-elected" hook
unit-admission-webhook-0: 15:50:14 INFO unit.admission-webhook/0.juju-log Generating certificates if missing.
unit-admission-webhook-0: 15:50:14 INFO unit.admission-webhook/0.juju-log Certificates already exist.
unit-admission-webhook-0: 15:50:14 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log Generating certificates if missing.
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log Certificates already exist.
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log HTTP Request: GET https://10.152.183.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log Rendering manifests
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-admin?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-edit?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-view?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:16 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:16 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:16 INFO unit.admission-webhook/0.juju-log Reconcile completed successfully
unit-admission-webhook-0: 15:50:16 INFO unit.admission-webhook/0.juju-log HTTP Request: GET https://10.152.183.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:16 INFO unit.admission-webhook/0.juju-log Rendering manifests
unit-admission-webhook-0: 15:50:17 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/poddefaults.kubeflow.org?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:17 INFO unit.admission-webhook/0.juju-log Reconcile completed successfully
unit-admission-webhook-0: 15:50:17 INFO unit.admission-webhook/0.juju-log Pebble plan updated with new configuration, replanning
unit-admission-webhook-0: 15:50:17 ERROR unit.admission-webhook/0.juju-log Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/charmed_kubeflow_chisme/pebble/_update_layer.py", line 31, in update_layer
    container.replan()
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/model.py", line 2252, in replan
    self._pebble.replan_services()
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/pebble.py", line 1993, in replan_services
    return self._services_action('replan', [], timeout, delay)
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/pebble.py", line 2090, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "base-admission-webhook" (cannot start service: exited quickly with code 0)
----- Logs from task 0 -----
2024-09-13T15:50:17Z INFO Service "admission-webhook" already started.
----- Logs from task 1 -----
2024-09-13T15:50:17Z INFO Most recent service output:
    {"level":"info","ts":"2024-09-13T15:50:17Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
    I0913 15:50:17.199753      32 main.go:771] About to start serving webhooks: &http.Server{Addr:":4443", Handler:http.Handler(nil), DisableGeneralOptionsHandler:false, TLSConfig:(*tls.Config)(0xc000502680), ReadTimeout:0, ReadHeaderTimeout:0, WriteTimeout:0, IdleTimeout:0, MaxHeaderBytes:0, TLSNextProto:map[string]func(*http.Server, *tls.Conn, http.Handler)(nil), ConnState:(func(net.Conn, http.ConnState))(nil), ErrorLog:(*log.Logger)(nil), BaseContext:(func(net.Listener) context.Context)(nil), ConnContext:(func(context.Context, net.Conn) context.Context)(nil), inShutdown:atomic.Bool{_:atomic.noCopy{}, v:0x0}, disableKeepAlives:atomic.Bool{_:atomic.noCopy{}, v:0x0}, nextProtoOnce:sync.Once{done:0x0, m:sync.Mutex{state:0, sema:0x0}}, nextProtoErr:error(nil), mu:sync.Mutex{state:0, sema:0x0}, listeners:map[*net.Listener]struct {}(nil), activeConn:map[*http.conn]struct {}(nil), onShutdown:[]func()(nil), listenerGroup:sync.WaitGroup{noCopy:sync.noCopy{}, state:atomic.Uint64{_:atomic.noCopy{}, _:atomic.align64{}, v:0x0}, sema:0x0}}
    {"level":"info","ts":"2024-09-13T15:50:17Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
2024-09-13T15:50:17Z ERROR cannot start service: exited quickly with code 0
-----

orfeas-k

Tested in CKF deployment and works

* Integrate rock * Update rock image * Update rock image * Uncomment test on test_charm.py * Update rock revision

Integrate rock

25ba857

mvlassis requested a review from a team as a code owner September 2, 2024 09:20

github-actions bot added the Libraries: Out of sync label Sep 2, 2024

mvlassis added 3 commits September 13, 2024 17:00

Update rock image

f434850

Update rock image

37b71a7

Uncomment test on test_charm.py

cb4c9a5

mvlassis mentioned this pull request Sep 16, 2024

Admission-webhook: Multiple Pebble services when integrated into Charmed Kubeflow canonical/kubeflow-rocks#136

Closed

Update rock revision

f6e36d2

orfeas-k approved these changes Sep 19, 2024

View reviewed changes

mvlassis merged commit f8f8e46 into main Sep 19, 2024
7 checks passed

mvlassis deleted the kf-6154-integrate-poddefaults-rocks branch September 19, 2024 10:04

mvlassis added a commit that referenced this pull request Sep 19, 2024

Integrate PodDefaults rock (#153)

87c19fa

* Integrate rock * Update rock image * Update rock image * Uncomment test on test_charm.py * Update rock revision

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate `PodDefaults` rock #153

Integrate `PodDefaults` rock #153

mvlassis commented Sep 2, 2024 •

edited

Loading

mvlassis commented Sep 13, 2024

orfeas-k left a comment

Integrate PodDefaults rock #153

Integrate PodDefaults rock #153

Conversation

mvlassis commented Sep 2, 2024 • edited Loading

mvlassis commented Sep 13, 2024

orfeas-k left a comment

Choose a reason for hiding this comment

Integrate `PodDefaults` rock #153

Integrate `PodDefaults` rock #153

mvlassis commented Sep 2, 2024 •

edited

Loading