[YUNIKORN-2504] Support canonical labels for queue/applicationId in scheduler

[chenyulin0719]

What is this PR for?

Support canonical Queue/ApplicationId labels in Pod, allows it coexist with the existing metadata.

• yunikorn.apache.org/app-id (New, Canonical Label)
• yunikorn.apache.org/queue (New, Canonical Label)

Check metadata consistency before move task state from 'New' to 'Pending',
Run the pod metadata check in task.sanityCheckBeforeScheduling()

• If sanity check is failed due to inconsistent metadata, move the task from 'New' to 'Rejected' state. And fail the pod with reasons. (Please check below screenshots.)

What type of PR is it?

• - Feature

Todos

Admission Controller should fail the pod request too if the metadata is inconsistent. Will create another Jira once this PR got approved.

What is the Jira issue?

https://issues.apache.org/jira/browse/YUNIKORN-2504

How should this be tested?

Run e2e test:

• (basic_scheduling) Verify_Pod_With_Conflicting_AppId
• (basic_scheduling) Verify_Pod_With_Conflicting_QueueName
• (recovery_and_restart) Verify_Pod_Restart_After_Add_Conflict_Metadata

Run below simple sleep pods:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: sleep
    yunikorn.apache.org/app-id: "application-sleep-0001"
    yunikorn.apache.org/queue: "root.sandbox"
  annotations:
    yunikorn.apache.org/queue: "root.sandbox-another"
  name: pod-with-inconsistent-queue
spec:
  schedulerName: yunikorn
  restartPolicy: Never
  containers:
    - name: sleep-6000s
      image: "alpine:latest"
      command: ["sleep", "6000"]
      resources:
        requests:
          cpu: "100m"
          memory: "500M"
          


---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: sleep
    yunikorn.apache.org/app-id: "application-sleep-0002"
  annotations:
    yunikorn.apache.org/app-id: "application-sleep-0002-another"
  name: pod-with-inconsistent-app-id
spec:
  schedulerName: yunikorn
  restartPolicy: Never
  containers:
    - name: sleep-6000s
      image: "alpine:latest"
      command: ["sleep", "6000"]
      resources:
        requests:
          cpu: "100m"
          memory: "500M"
          
--- 
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: sleep
    yunikorn.apache.org/app-id: "application-sleep-0003"
    yunikorn.apache.org/queue: "root.sandbox"
  name: pod-with-correct-labels
spec:
  schedulerName: yunikorn
  restartPolicy: Never
  containers:
    - name: sleep-6000s
      image: "alpine:latest"
      command: ["sleep", "6000"]
      resources:
        requests:
          cpu: "100m"
          memory: "500M"
---

Screenshots (if appropriate)

Below is the screenshot without admission controller:

Questions:

NA

[chenyulin0719]

To make it easier for the reviewer to read, add some notes.

1. Can start from 'task.sanityCheckBeforeScheduling()' in applications.go
2. Can check e2e tests in basic_scheduling_test and recover_and_restart first.

[chenyulin0719]

Perform a sanity check before move this task to Pending state.

Before this PR, sanity check only check PVC's readiness

• If sanity check passed, move task state from 'New' -> 'Pending'
• If sanity check failed, task state remains in 'New' (Will be checked again in next schedule cycle)

After this PR (Sanity check check PVC and Pod Metadata)

• if sanity check passed, 'New' -> 'Pending'
• if sanity check fails due to PVC -> 'New' (No change)
• if sanity check fails due to a unbound pod with inconsistent metadata (AppID/Label), move task state from 'New' to 'Rejected'

Design decision: Only reject unbound pods because we don't want to failed existing running pod after restart YK.

[chenyulin0719]

Moved failTaskPodWithReasonAndMsg() to task.go

change

• podCopy := task.GetTaskPod().DeepCopy()
  to
• podCopy := task.pod.DeepCopy()

to prevent deadlock when task state machine is handling TaskRejected event. (WLock)

[chenyulin0719]

Note:

• We can directly use canonical representation for placeholder here. The newer version shim allows legacy and canonical representation metadata coexists.

[chenyulin0719]

Design decision:

• Fail the pod if the task's reject reason is inconsistent metadata (PodInconsistentMetadata).

[chenyulin0719]

Mark this PR as draft after the discussion of the community sync-up.

To be changed:

1. Ordering change to (Cannonical Label -> Annotation -> Queue(Existing))
2. Version 1.6.0 only prints out warnings; task rejection should occur in version 1.7.0.

[chenyulin0719]

Close this draft PR and another PR for formal review was created.
#871