Merge branch 'master' into YUNIKORN-1998

apache · Oct 30, 2023 · 2bc9d9d · 2bc9d9d
2 parents 45417fe + 6a30d08
commit 2bc9d9d
Show file tree

Hide file tree

Showing 64 changed files with 497 additions and 925 deletions.
diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml
@@ -4,6 +4,7 @@ on:
   pull_request:
     branches:
       - master
+  workflow_dispatch: {}
 
 jobs:
   build:

diff --git a/Makefile b/Makefile
@@ -510,7 +510,7 @@ bench:
 fsm_graph:
 	@echo "generating FSM graphs"
 	"$(GO)" clean -testcache
-	"$(GO)" test -tags graphviz -run 'Test.*FsmGraph' ./pkg/shim ./pkg/cache
+	"$(GO)" test -tags graphviz -run 'Test.*FsmGraph' ./pkg/cache
 	scripts/generate-fsm-graph-images.sh
 
 # Remove generated build artifacts

diff --git a/deployments/examples/authz/README.md b/deployments/examples/authz/README.md
@@ -23,25 +23,31 @@ Yunikorn offers a range of features, including advanced capabilities like hierar
 The following will be included in this article：
 
 - [Access control with ACL](./acl)
-- [Placement of different users](./placementRule)
-- [Limit usable resources on a queue level](./resourceLimit)
+- [Placement of different users](./placement-rules)
+- [Limit usable resources on a queue level](./resource-limits)
 - [Preemption and priority scheduling with fencing](./priority)
 
 ## Prerequisites
 
 Before configuring yunikorn-config, we need to create users using [Authentication](https://kubernetes.io/docs/reference/access-authn-authz/authentication/) and [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) from Kubernetes.
 
-To create the necessary users for the examples, Please use [./create-user.sh](./k8s-api-access/create-user.sh) to create a user.
+To create the necessary users for the examples, first navigate to the `k8s-api-access` directory:
+
+```shell
+cd k8s-api-access
+```
+
+Then use [./create-user.sh](./k8s-api-access/create-user.sh) to create a user.
 
 After the user is created, the pod can be obtained by the following command to confirm the creation is successful：
 
-```yaml
+```shell
 kubectl --context=sue-context get pod
 ```
 
 In our use cases, we frequently simulate different users deploying YAML files. To accomplish this, we utilize the `--context` command to select the appropriate user for each deployment:
 
-```yaml
+```shell
 kubectl --context=sue-context apply -f ./acl/nginx-1.yaml
 ```
 
@@ -83,23 +89,23 @@ The following example illustrates this scenario, along with the expected test re
 
 ## Placement of different users
 
-In [yunikorn-configs.yaml](./placementRule/yunikorn-configs.yaml), we use `placementrules` to allow the scheduler to dynamically assign applications to a queue, and even create a new queue if needed.
+In [yunikorn-configs.yaml](./placement-rules/yunikorn-configs.yaml), we use `placementrules` to allow the scheduler to dynamically assign applications to a queue, and even create a new queue if needed.
 
 See the documentation on [App Placement Rules](https://yunikorn.apache.org/docs/user_guide/placement_rules) for more information.
 
 ```yaml
 placementrules:
   - name: provided
-  create: true
-  filter:
-    type: allow
-    users:
-      - admin
-    groups:
-      - admin
-  parent:
-    name: fixed
-    value: root.system
+    create: true
+    filter:
+      type: allow
+      users:
+        - admin
+      groups:
+        - admin
+    parent:
+      name: fixed
+      value: root.system
 ```
 
 In the test case, the user doesn't need to specify the queue for their application. Instead, the scheduler will utilize the placement rules to assign the application to the appropriate queue. If needed, the scheduler will create new queues.
@@ -108,15 +114,15 @@ The following example illustrates this scenario, along with the expected test re
 
 | placement rule         | user, group  | provide queue             | namespace | Expected to be placed on  | YAML filename                                        |
 |------------------------|--------------|---------------------------|-----------|---------------------------|------------------------------------------------------|
-| provided               | admin, admin | root.system.high-priority |           | root.system.high-priority | [nginx-admin.yaml](./placementRule/nginx-admin.yaml) |
-| provided               | admin, admin | root.system.low-priority  |           | root.system.low-priority  | [nginx-admin.yaml](./placementRule/nginx-admin.yaml) |
-| username               | sue, group-a |                           |           | root.tenants.group-a.sue  | [nginx-sue.yaml](./placementRule/nginx-sue.yaml)     |
-| tag (value: namespace) | kim, group-b |                           | dev       | root.tenants.group-b.dev  | [nginx-kim.yaml](./placementRule/nginx-kim.yaml)     |
-| tag (value: namespace) | kim, group-b |                           | test      | root.tenants.group-b.test | [nginx-kim.yaml](./placementRule/nginx-kim.yaml)     |
+| provided               | admin, admin | root.system.high-priority |           | root.system.high-priority | [nginx-admin.yaml](./placement-rules/nginx-admin.yaml) |
+| provided               | admin, admin | root.system.low-priority  |           | root.system.low-priority  | [nginx-admin.yaml](./placement-rules/nginx-admin.yaml) |
+| username               | sue, group-a |                           |           | root.tenants.group-a.sue  | [nginx-sue.yaml](./placement-rules/nginx-sue.yaml)     |
+| tag (value: namespace) | kim, group-b |                           | dev       | root.tenants.group-b.dev  | [nginx-kim.yaml](./placement-rules/nginx-kim.yaml)     |
+| tag (value: namespace) | kim, group-b |                           | test      | root.tenants.group-b.test | [nginx-kim.yaml](./placement-rules/nginx-kim.yaml)     |
 
 ## Limit usable resources on a queue level
 
-In [yunikorn-configs.yaml](./resourceLimit/yunikorn-configs.yaml), we use `resources` to limit and reserve the amount of resources per queue.
+In [yunikorn-configs.yaml](./resource-limits/yunikorn-configs.yaml), we use `resources` to limit and reserve the amount of resources per queue.
 
 See the documentation on [Partition and Queue Configuration #Resources](https://yunikorn.apache.org/docs/user_guide/queue_config#resources) for more information.
 
@@ -137,12 +143,12 @@ The following example illustrates this scenario, along with the expected test re
 
 | user, group  | Resource Limits for Destination Queues | request resources for each replicas | replica | result                                                   | YAML filename                                        |
 |--------------|----------------------------------------|-------------------------------------|---------|----------------------------------------------------------|------------------------------------------------------|
-| admin, admin | {memory: 6G, vcore: 6}                 | {memory: 512M, vcore: 250m}         | 1       | run all replica                                          | [nginx-admin.yaml](./resourceLimit/nginx-admin.yaml) |
-| sue, group-A | {memory: 2G, vcore: 4}                 | {memory: 512M, vcore: 500m}         | 5       | run 3 replica (4 replica will exceed the resource limit) | [nginx-sue.yaml](./resourceLimit/nginx-sue.yaml)     |
+| admin, admin | {memory: 6G, vcore: 6}                 | {memory: 512M, vcore: 250m}         | 1       | run all replica                                          | [nginx-admin.yaml](./resource-limits/nginx-admin.yaml) |
+| sue, group-A | {memory: 2G, vcore: 4}                 | {memory: 512M, vcore: 500m}         | 5       | run 3 replica (4 replica will exceed the resource limit) | [nginx-sue.yaml](./resource-limits/nginx-sue.yaml)     |
 
 ## Preemption and priority scheduling with fencing
 
-In [yunikorn-configs.yaml](./resourceLimit/yunikorn-configs.yaml), we use `priority.offset` and `priority.policy` to configure the priority in a queue.
+In [yunikorn-configs.yaml](./resource-limits/yunikorn-configs.yaml), we use `priority.offset` and `priority.policy` to configure the priority in a queue.
 
 See the documentation on [App & Queue Priorities](https://yunikorn.apache.org/docs/user_guide/priorities) for more information.
 

diff --git a/deployments/examples/placements/README.md b/deployments/examples/placements/README.md
@@ -28,12 +28,12 @@ The sleep pod is described in the example file and the partition is provided in
 
 ## How to adopt the queues in `config.yaml`
 Before deploying the pods, make sure that the data in the `yunikorn-configs` configmap is correct.
-For example, `queue.yaml` in the `yunikorn-configs` configmap should be updated before starting fixed example.
+For example, `queues.yaml` in the `yunikorn-configs` configmap should be updated before starting fixed example.
 `yunikorn-configs` configmap should contains following information.
----
-**_NOTE:_** 
-`queue.yaml` should be __full__ queue config and then legal configuration would be updated to Yunikorn.
----
+
+> **_NOTE:_**
+> `queues.yaml` should be __full__ queue config and then legal configuration would be updated to Yunikorn.
+
 ```
 yunikornDefaults:
   queues.yaml: |

diff --git a/deployments/examples/placements/fixed/config.yaml b/deployments/examples/placements/fixed/config.yaml
@@ -15,8 +15,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Update this queue yaml to the yunikorn-configs configmap before trying this example.
-queue.yaml: |
+# Update the queues.yaml to the yunikorn-configs configmap before trying this example.
+queues.yaml: |
   partitions:
     - name: default
       placementrules:
@@ -27,4 +27,3 @@ queue.yaml: |
         submitacl: '*'
         queues:
         - name: last_resort
-
diff --git a/deployments/examples/placements/fixed/fixed_example.yaml b/deployments/examples/placements/fixed/fixed_example.yaml
@@ -20,9 +20,10 @@ kind: Pod
 metadata:
   labels:
     app: sleep
-    applicationId: "prvoided-rule-example01"
+    applicationId: "fixed-rule-example01"
     queue: "my_special_queue"
-    yunikorn.apache.org/username: developer
+  annotations:
+    yunikorn.apache.org/user.info: "{\"user\": \"developer\"}"
   name: task0
 spec:
   schedulerName: yunikorn
@@ -34,4 +35,3 @@ spec:
         requests:
           cpu: "100m"
           memory: "500M"
-
diff --git a/deployments/examples/placements/provided/config.yaml b/deployments/examples/placements/provided/config.yaml
@@ -15,8 +15,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Update this queue yaml to the yunikorn-configs configmap before trying this example.
-queue.yaml: |
+# Update the queues.yaml to the yunikorn-configs configmap before trying this example.
+queues.yaml: |
   partitions:
     - name: default
       placementrules:
@@ -28,4 +28,3 @@ queue.yaml: |
       queues:
       - name: root
         submitacl: '*'
-
diff --git a/deployments/examples/placements/provided/provided_example.yaml b/deployments/examples/placements/provided/provided_example.yaml
@@ -22,7 +22,8 @@ metadata:
     app: sleep
     applicationId: "prvoided-rule-example01"
     queue: "my_special_queue"
-    yunikorn.apache.org/username: "developer"
+  annotations:
+    yunikorn.apache.org/user.info: "{\"user\": \"developer\"}"
   name: task0
 spec:
   schedulerName: yunikorn
@@ -34,4 +35,3 @@ spec:
         requests:
           cpu: "100m"
           memory: "500M"
-
diff --git a/deployments/examples/placements/tag/config.yaml b/deployments/examples/placements/tag/config.yaml
@@ -15,8 +15,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Update this queue yaml to the yunikorn-configs configmap before trying this example.
-queue.yaml: |
+# Update the queues.yaml to the yunikorn-configs configmap before trying this example.
+queues.yaml: |
   partitions:
     - name: default
       placementrules:
@@ -26,4 +26,3 @@ queue.yaml: |
       queues:
       - name: root
         submitacl: '*'
-
diff --git a/deployments/examples/placements/tag/tag_example.yaml b/deployments/examples/placements/tag/tag_example.yaml
@@ -20,8 +20,9 @@ kind: Pod
 metadata:
   labels:
     app: sleep
-    applicationId: "prvoided-rule-example01"
-    yunikorn.apache.org/username: developer
+    applicationId: "tag-rule-example01"
+  annotations:
+    yunikorn.apache.org/user.info: "{\"user\": \"developer\"}"
   name: task0
 spec:
   schedulerName: yunikorn
@@ -40,8 +41,9 @@ metadata:
   namespace: testing
   labels:
     app: sleep
-    applicationId: "prvoided-rule-example02"
-    yunikorn.apache.org/username: developer
+    applicationId: "tag-rule-example02"
+  annotations:
+    yunikorn.apache.org/user.info: "{\"user\": \"developer\"}"
   name: task1
 spec:
   schedulerName: yunikorn
@@ -53,4 +55,3 @@ spec:
         requests:
           cpu: "100m"
           memory: "500M"
-
diff --git a/deployments/examples/placements/username/config.yaml b/deployments/examples/placements/username/config.yaml
@@ -15,8 +15,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Update this queue yaml to the yunikorn-configs configmap before trying this example.
-queue.yaml: |
+# Update the queues.yaml to the yunikorn-configs configmap before trying this example.
+queues.yaml: |
   partitions:
     - name: default
       placementrules:
@@ -27,4 +27,3 @@ queue.yaml: |
         submitacl: '*'
         queues:
         - name: finance_dot_test
-
diff --git a/deployments/examples/placements/username/username_example.yaml b/deployments/examples/placements/username/username_example.yaml
@@ -21,7 +21,8 @@ metadata:
   labels:
     app: sleep
     applicationId: "username-rule-example01"
-    yunikorn.apache.org/username: finance.test
+  annotations:
+    yunikorn.apache.org/user.info: "{\"user\": \"finance.test\"}"
   name: task0
 spec:
   schedulerName: yunikorn
@@ -40,7 +41,8 @@ metadata:
   labels:
     app: sleep
     applicationId: "username-rule-example02"
-    yunikorn.apache.org/username: developer
+  annotations:
+    yunikorn.apache.org/user.info: "{\"user\": \"developer\"}"
   name: task1
 spec:
   schedulerName: yunikorn
@@ -52,4 +54,3 @@ spec:
         requests:
           cpu: "100m"
           memory: "500M"
-
diff --git a/go.mod b/go.mod
@@ -22,8 +22,8 @@ go 1.20
 
 require (
 	github.com/GoogleCloudPlatform/spark-on-k8s-operator v0.0.0-20201215015655-2e8b733f5ad0
-	github.com/apache/yunikorn-core v0.0.0-20230914150517-9dd3df7dadb2
-	github.com/apache/yunikorn-scheduler-interface v0.0.0-20230914150238-ff06224fc98d
+	github.com/apache/yunikorn-core v0.0.0-20231025072825-bc7c00124c4c
+	github.com/apache/yunikorn-scheduler-interface v0.0.0-20231020041412-6f80d179257c
 	github.com/google/go-cmp v0.5.9
 	github.com/google/uuid v1.3.0
 	github.com/looplab/fsm v1.0.1
@@ -52,7 +52,7 @@ require (
 	github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/blang/semver/v4 v4.0.0 // indirect
-	github.com/cenkalti/backoff/v4 v4.1.3 // indirect
+	github.com/cenkalti/backoff/v4 v4.2.1 // indirect
 	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/coreos/go-semver v0.3.0 // indirect
 	github.com/coreos/go-systemd/v22 v22.4.0 // indirect
@@ -80,7 +80,7 @@ require (
 	github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
 	github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc // indirect
 	github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
-	github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.16.0 // indirect
 	github.com/imdario/mergo v0.3.7 // indirect
 	github.com/inconshreveable/mousetrap v1.0.1 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
@@ -115,31 +115,32 @@ require (
 	go.etcd.io/etcd/client/pkg/v3 v3.5.7 // indirect
 	go.etcd.io/etcd/client/v3 v3.5.7 // indirect
 	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.35.0 // indirect
-	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.35.1 // indirect
-	go.opentelemetry.io/otel v1.10.0 // indirect
-	go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.10.0 // indirect
-	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.10.0 // indirect
-	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.10.0 // indirect
-	go.opentelemetry.io/otel/metric v0.31.0 // indirect
-	go.opentelemetry.io/otel/sdk v1.10.0 // indirect
-	go.opentelemetry.io/otel/trace v1.10.0 // indirect
-	go.opentelemetry.io/proto/otlp v0.19.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.44.0 // indirect
+	go.opentelemetry.io/otel v1.19.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.19.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.19.0 // indirect
+	go.opentelemetry.io/otel/metric v1.19.0 // indirect
+	go.opentelemetry.io/otel/sdk v1.19.0 // indirect
+	go.opentelemetry.io/otel/trace v1.19.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.0.0 // indirect
 	go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5 // indirect
 	go.uber.org/atomic v1.7.0 // indirect
 	go.uber.org/multierr v1.6.0 // indirect
-	golang.org/x/crypto v0.13.0 // indirect
-	golang.org/x/net v0.15.0 // indirect
-	golang.org/x/oauth2 v0.7.0 // indirect
+	golang.org/x/crypto v0.14.0 // indirect
+	golang.org/x/net v0.17.0 // indirect
+	golang.org/x/oauth2 v0.10.0 // indirect
 	golang.org/x/sync v0.3.0 // indirect
-	golang.org/x/sys v0.12.0 // indirect
-	golang.org/x/term v0.12.0 // indirect
+	golang.org/x/sys v0.13.0 // indirect
+	golang.org/x/term v0.13.0 // indirect
 	golang.org/x/text v0.13.0 // indirect
 	golang.org/x/time v0.3.0 // indirect
 	golang.org/x/tools v0.13.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
-	google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect
-	google.golang.org/grpc v1.56.0 // indirect
-	google.golang.org/protobuf v1.30.0 // indirect
+	google.golang.org/genproto v0.0.0-20230711160842-782d3b101e98 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20230711160842-782d3b101e98 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20230711160842-782d3b101e98 // indirect
+	google.golang.org/grpc v1.58.3 // indirect
+	google.golang.org/protobuf v1.31.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
@@ -163,10 +164,10 @@ require (
 )
 
 replace (
-	golang.org/x/crypto => golang.org/x/crypto v0.13.0
+	golang.org/x/crypto => golang.org/x/crypto v0.14.0
 	golang.org/x/lint => golang.org/x/lint v0.0.0-20210508222113-6edffad5e616
-	golang.org/x/net => golang.org/x/net v0.15.0
-	golang.org/x/sys => golang.org/x/sys v0.12.0
+	golang.org/x/net => golang.org/x/net v0.17.0
+	golang.org/x/sys => golang.org/x/sys v0.13.0
 	golang.org/x/text => golang.org/x/text v0.13.0
 	golang.org/x/tools => golang.org/x/tools v0.13.0
 	k8s.io/api => k8s.io/api v0.27.3
-Original file line number
+Diff line change
@@ Expand Up / @@ -4,6 +4,7 @@ on: @@
       pull_request:
         branches:
           - master
+      workflow_dispatch: {}
     jobs:
       build:
@@ Expand Down @@