Skip to inject the metrics-collector pods to the katib controller

[tenzen-y]

What this PR does / why we need it:
I introduced the label to skip injecting the metrics-collector to the katib system components.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2069

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tenzen-y]

I discussed offline with @andreyvelich and @johnugeorge.

Let me summarize. The replicaset controller occasionally faces the following errors since the webhooks are registered to the kube-apiserver before the controller pod is started.

Warning  FailedCreate  39s (x15 over 2m2s)  replicaset-controller  Error creating: Internal error occurred: failed calling webhook "mutator.pod.katib.kubeflow.org": failed to call webhook: Post "[https://katib-controller.kubeflow.svc:443/mutate-pod?timeout=10s](https://katib-controller.kubeflow.svc/mutate-pod?timeout=10s)": dial tcp 10.100.157.27:443: connect: connection refused

So we set the label to avoid injecting the metrics-collector pod only into the controller pod.

Also in the future, we will switch to AdmissionWebhookMatchConditions feature to process requests more accurately once the feature gate is enabled as a default.

[tenzen-y]

/assign @andreyvelich @johnugeorge

[andreyvelich]

Thank you @tenzen-y!
/lgtm
/hold for review
/assign @johnugeorge @gaocegege

[johnugeorge]

/hold cancel