[YUNIKORN-1452] document admission controller use case #369

youxuan0714 · 2023-11-17T09:02:21Z

What is this PR for?

Add and integrate documents about admission controller

What type of PR is it?

Todos

- Task

What is the Jira issue?

open an issue on Jira https://issues.apache.org/jira/browse/YUNIKORN-1452

How should this be tested?

Screenshots (if appropriate)

Questions:

- The licenses files need update.
- There is breaking changes for older versions.
- It needs documentation.

wilfred-s

Please do not mix changes in one PR (remove queue.md)
Do not commit/update the package.json file unless we change the docusaurus release.

youxuan0714 · 2023-11-21T08:45:16Z

@wilfred-s, thanks for reminding, I have corrected it.

wilfred-s

This will probably become much clearer if we point to a use case to make clear when what and how:
use case 1: standard deployment, everything scheduled by YuniKorn
use case 2: standard deployment, user details from
use case 3: plugin deployment everything through YuniKorn
use case 4: plugin deployment, only workloads in some namespaces scheduled by YuniKorn
We can then hook the different settings into those and explain how things work

wilfred-s · 2023-11-29T05:01:16Z

docs/user_guide/admission_controller.md

+| Variable                  | Default value                           | Description                                                                |
+|---------------------------|-----------------------------------------|----------------------------------------------------------------------------|


These are already part of the Service Configuration. We should not have the same information in two locations as that becomes a sync issue. Better is to point back to the service config.

wilfred-s · 2023-11-29T05:18:39Z

docs/user_guide/admission_controller.md

+Access control configuration, all entries start with the prefix `admissionController.accessControl.`.
+
+| Variable           | Default value                         | Description                                                                |


These are already part of the Service Configuration. We should not have the same information in two locations as that becomes a sync issue. Better is to point back to the service config.

wilfred-s · 2023-11-29T05:21:27Z

docs/user_guide/admission_controller.md

+The admission controller runs in a separate pod, it runs a [mutation webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook)
+and a [validation webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook), where:
+
+1. The `mutation webhook` mutates pod spec by:


make separate heading i.e. ###

also missing the fact that pod updates are checked for changes to the user annotations: no changes allowed for already created pods.

wilfred-s · 2023-11-29T05:21:59Z

docs/user_guide/admission_controller.md

+        - Otherwise, adds `queue: root.default`
+    - Adding `disableStateAware` label
+        - If pod was assigned a generated applicationId by the admission controller, also set `disableStateAware: true`. This causes the generated application to immediately transition from the `Starting` to `Running` state so that it will not block other applications.
+2. The `validation webhook` validates the configuration set in the configmap


make a separate heading i.e. ###

wilfred-s · 2023-11-29T05:27:49Z

docs/user_guide/admission_controller.md

+    - Adding `disableStateAware` label
+        - If pod was assigned a generated applicationId by the admission controller, also set `disableStateAware: true`. This causes the generated application to immediately transition from the `Starting` to `Running` state so that it will not block other applications.
+2. The `validation webhook` validates the configuration set in the configmap
+    - This is used to prevent writing malformed configuration into the configmap.


Explain in text instead of bullet points

wilfred-s · 2023-11-29T05:29:29Z

docs/user_guide/admission_controller.md

+and a [validation webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook), where:
+
+1. The `mutation webhook` mutates pod spec by:
+    - Adding `schedulerName: yunikorn`


Use text to explain and link back to the configuration

wilfred-s · 2023-11-29T05:33:09Z

docs/user_guide/admission_controller.md

+
+On startup, the admission controller performs a series of tasks to ensure that it is properly registered with Kubernetes:
+
+1. Loads a Kubernetes secret called `admission-controller-secrets`. This secret stores a pair of CA certificates which are used to sign the TLS server certificate used by the admission controller.


Explain in text instead of bullet points.

youxuan0714 · 2023-12-25T14:45:31Z

Hi @wilfred-s, sorry it took so long. I've modified it as suggested, but there are still some things I'm not sure about, for example, the 'user detail from' in use case 2. Please help confirm what else needs to be modified or added, thank you.

pbacsko

Found some typo & grammar issues spotted by Google docs.

pbacsko · 2024-01-25T09:35:01Z

docs/user_guide/admission_controller.md

+
+In addition, The `mutation webhook` will also add the following labels:
+- `applicationId` label. If applicationId or spark-app-selector already exists, it will be used. Otherwise, an applicationId will be generated for the pod. The default is `yunikorn-<namespace>-autogen`. Pods in the same namespace will generate the same applicationId. For other generation methods, please refer to [Service configuration](service_config.md). 
+- `queue` label. If there is a queue label, it will be used. Note that if placement rule is enabled, the value set in label will be ignored. Otherwise, `root.default` will be assigned by default. If you want to change the default queue name, please refer to [Service configuration](service_config.md).


nit: "if placement rules are enabled, the value set in the label will be ignored"

pbacsko · 2024-01-25T09:35:36Z

docs/user_guide/admission_controller.md

+On startup, the admission controller performs a series of tasks to ensure that it is properly registered with Kubernetes:
+
+Loads a Kubernetes secret called `admission-controller-secrets`. This secret stores a pair of CA certificates which are used to sign the TLS server certificate used by the admission controller.
+If the secret cannot be found or either CA certificate is within 90 days of expiration, generates new certificate(s). If a certificate is expiring, a new one is generated with an expiration of 12 months in the future. If both certificates are missing or expiring, the second certificate is generated with an expiration of 6 months in the future. This ensures that both certificates do not expire at the same time, and that there is an overlap of trusted certificates.


nit: "it generates new certificate(s)"

pbacsko · 2024-01-25T09:36:55Z

docs/user_guide/admission_controller.md

+If the CA certificates were created or updated, writes the secrets back to Kubernetes.
+Generates an ephemeral TLS server certificate signed by the CA certificate with the latest expiration date.
+Validates, and if necessary, creates or updates the Kubernetes webhook configurations named `yunikorn-admission-controller-validations` and `yunikorn-admission-controller-mutations`. If the CA certificates have changed, the webhooks will also be updated. These webhooks allow the Kubernetes API server to connect to the admission controller service to perform configmap validations and pod mutations.
+Starts up the admission controller HTTPS server.


nit: missing subject (what starts up the https server?).

pbacsko · 2024-01-25T09:37:28Z

docs/user_guide/admission_controller.md

+
+Loads a Kubernetes secret called `admission-controller-secrets`. This secret stores a pair of CA certificates which are used to sign the TLS server certificate used by the admission controller.
+If the secret cannot be found or either CA certificate is within 90 days of expiration, generates new certificate(s). If a certificate is expiring, a new one is generated with an expiration of 12 months in the future. If both certificates are missing or expiring, the second certificate is generated with an expiration of 6 months in the future. This ensures that both certificates do not expire at the same time, and that there is an overlap of trusted certificates.
+If the CA certificates were created or updated, writes the secrets back to Kubernetes.


nit: "it writes"

pbacsko · 2024-01-25T09:38:40Z

docs/user_guide/admission_controller.md

+
+For more details on admission controller configuration, please refer to [Service configuration](service_config.md).
+
+For more information on user and group access management, please view [User&Group Resolutiona](usergroup_resolution.md).


Typo: "resolutiona"

pbacsko · 2024-01-25T09:39:23Z

docs/user_guide/admission_controller.md

+```
+
+### Use case
+The following are the usage scenarios of using admission controller to set up in different deployment modes of Yunikorn.


nit: "of using the"

pbacsko · 2024-01-25T09:39:38Z

docs/user_guide/admission_controller.md

+#### Standard deployment, everything scheduled by Yunikorn
+In standard deployment mode, can set which namespaces will be scheduled by YuniKorn through the `admissionController.filtering.processNamespaces` tag in `yunikorn-config`.
+
+If want all namespaces to be scheduled by YuniKorn, just set `admissionController.filtering.processNamespaces` to the empty string (default).


"If you want"

pbacsko · 2024-01-25T09:40:04Z

docs/user_guide/admission_controller.md

+#### Plugin deployment, everything scheduled by YuniKorn
+Same as the standard mode, can set which namespaces will be scheduled by YuniKorn through the `admissionController.filtering.processNamespaces` tag in `yunikorn-config`.
+
+If want all namespaces to be scheduled by YuniKorn, just set `admissionController.filtering.processNamespaces` to the empty string (default).


"If you want"

pbacsko · 2024-01-25T09:42:03Z

docs/user_guide/admission_controller.md

+```
+
+#### Plugin deployment, everything scheduled by YuniKorn
+Same as the standard mode, can set which namespaces will be scheduled by YuniKorn through the `admissionController.filtering.processNamespaces` tag in `yunikorn-config`.


"Just like in standard mode, the admissionController.filtering.processNamespaces tag sets which namespaces will be scheduled by Yunikorn..." or sth like that.

pbacsko · 2024-01-25T09:43:17Z

docs/user_guide/admission_controller.md

+#All namespaces will be schduler by YuniKorn
+admissionController.filtering.processNamespaces: ""
+```
+#### Plugin deployment, only workloads in some namespaces scheduled byYuniKorn


Typo: "byYunikorn"

I think the "only" is not needed here (sounds weird to me).

chia7712

@youxuan0714 thanks for nice patch. a couple of comments left.

chia7712 · 2024-01-27T15:20:59Z

docs/user_guide/admission_controller.md

+		spec:
+			container:
+				- name: nginx
+					image: "nginx:stable-alpine


there are some invalid format, please fix it.

chia7712 · 2024-01-27T15:44:34Z

docs/user_guide/admission_controller.md

+### Mutation webhook
+The `mutation webhook` will add `schedulerName:yunikorn` to the pod spec so that the pod can be scheduled by the Yunikorn scheduler.
+
+In addition, The `mutation webhook` will also add the following labels:


could you include the annotation also? https://github.com/apache/yunikorn-k8shim/blob/master/pkg/common/constants/constants.go#L95

chia7712 · 2024-01-27T15:50:36Z

docs/user_guide/admission_controller.md

+- `queue` label. If there is a queue label, it will be used. Note that if placement rule is enabled, the value set in label will be ignored. Otherwise, `root.default` will be assigned by default. If you want to change the default queue name, please refer to [Service configuration](service_config.md).
+- `disableStateAware` label, if the admission controller generates an applicationId for the pod, `disableStateAware: true` must also be set. This will cause the application to immediately transition from "Starting" to "Running" state so that it will not block other applications
+
+When updating a created pod, the admission controller will check whether the userinfo annotation before and after the update is the same. If it is different, the change will not be allowed.


pardon me, could you share reference to me? did not noticed this functionality before.

Checked the current AM implementation, it seems like the "check" behavior wasn't implemented.
And I don't think we need it too because AM won't generate new user info if user annotation has already been set.

https://github.com/apache/yunikorn-k8shim/blob/master/pkg/admission/admission_controller.go#L176

@chenyulin0719 thanks for confirmation.

It seems the related code was committed by apache/yunikorn-k8shim#469. @pbacsko Could you take a look? We can just delete the statement from the docs, or we can have a ticket to implement that behavior for AM

chia7712 · 2024-01-27T15:52:26Z

docs/user_guide/admission_controller.md

+The `validation webhook` calls scheduler [validation REST API](https://yunikorn.apache.org/docs/api/scheduler#configuration-validation) to validate configmap updates.
+
+## Admission controller deployment
+By default, the admission controller is deployed as part of the YuniKorn Helm chart installation. This can be disabled if necessary (though not recommended) by setting the Helm parameter `embedAdmissionController` to `false`.


"necessary (though not recommended)" could we have a section to discuss the reasons why we discourage users from doing this?

pbacsko · 2024-04-19T11:13:26Z

ping @youxuan0714 - do you have time to do the requested changes?

wilfred-s force-pushed the master branch from 6d7e69c to 778ca9e Compare November 20, 2023 04:28

wilfred-s assigned youxuan0714 Nov 21, 2023

wilfred-s requested changes Nov 21, 2023

View reviewed changes

youxuan0714 force-pushed the YUNIKORN-1452 branch from 1a39c9f to a9b2fac Compare November 21, 2023 06:17

YUNIKORN-1452

55ae98a

youxuan0714 force-pushed the YUNIKORN-1452 branch from a9b2fac to 55ae98a Compare November 21, 2023 06:51

wilfred-s requested changes Nov 29, 2023

View reviewed changes

youxuan0714 and others added 3 commits December 25, 2023 21:54

Merge branch 'apache:master' into YUNIKORN-1452

7897188

[YUNIKORN-1452] document admission controller use case

f6dceb3

Merge remote-tracking branch 'origin/YUNIKORN-1452' into YUNIKORN-1452

c3febaf

wilfred-s requested review from wilfred-s, FrankYang0529, craigcondit and pbacsko January 25, 2024 05:51

pbacsko requested changes Jan 25, 2024

View reviewed changes

chia7712 reviewed Jan 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[YUNIKORN-1452] document admission controller use case #369

[YUNIKORN-1452] document admission controller use case #369

youxuan0714 commented Nov 17, 2023

wilfred-s left a comment

youxuan0714 commented Nov 21, 2023

wilfred-s left a comment

wilfred-s Nov 29, 2023

wilfred-s Nov 29, 2023

wilfred-s Nov 29, 2023

wilfred-s Nov 29, 2023

wilfred-s Nov 29, 2023

wilfred-s Nov 29, 2023

wilfred-s Nov 29, 2023

youxuan0714 commented Dec 25, 2023

pbacsko left a comment

pbacsko Jan 25, 2024

pbacsko Jan 25, 2024

pbacsko Jan 25, 2024

pbacsko Jan 25, 2024

pbacsko Jan 25, 2024

pbacsko Jan 25, 2024

pbacsko Jan 25, 2024

pbacsko Jan 25, 2024

pbacsko Jan 25, 2024

pbacsko Jan 25, 2024

chia7712 left a comment

chia7712 Jan 27, 2024

chia7712 Jan 27, 2024

chia7712 Jan 27, 2024

chenyulin0719 Jan 29, 2024

chia7712 Jan 29, 2024

chia7712 Jan 27, 2024

pbacsko commented Apr 19, 2024

		\| Variable \| Default value \| Description \|
		\|---------------------------\|-----------------------------------------\|----------------------------------------------------------------------------\|

		Access control configuration, all entries start with the prefix `admissionController.accessControl.`.

		\| Variable \| Default value \| Description \|


		On startup, the admission controller performs a series of tasks to ensure that it is properly registered with Kubernetes:

		1. Loads a Kubernetes secret called `admission-controller-secrets`. This secret stores a pair of CA certificates which are used to sign the TLS server certificate used by the admission controller.


		For more details on admission controller configuration, please refer to [Service configuration](service_config.md).

		For more information on user and group access management, please view [User&Group Resolutiona](usergroup_resolution.md).

[YUNIKORN-1452] document admission controller use case #369

Are you sure you want to change the base?

[YUNIKORN-1452] document admission controller use case #369

Conversation

youxuan0714 commented Nov 17, 2023

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

wilfred-s left a comment

Choose a reason for hiding this comment

youxuan0714 commented Nov 21, 2023

wilfred-s left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

youxuan0714 commented Dec 25, 2023

pbacsko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pbacsko commented Apr 19, 2024