Update webhook failurePolicy from Ignore to Fail

[knkski]

What this PR does / why we need it:

Updates katib-controller webhook for operator and manifests to set failurePolicy to Fail. This allows for greater visibility on errors. Also updates webhook creation handling in katib-controller operator.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: knkski
To complete the pull request process, please assign gaocegege after the PR has been reviewed.
You can assign the PR to them by writing /assign @gaocegege in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coveralls]

Coverage decreased (-0.05%) to 74.088% when pulling 68e2761 on knkski:webhook-failure-policy into f5abfd0 on kubeflow:master.

[gaocegege]

/cc @andreyvelich

[andreyvelich]

Thank you for this update @knkski!
Do we want to include this change to Katib 0.13 release ?

[andreyvelich]

/retest

[andreyvelich]

It seems that this change doesn't work on K8s 1.18.
@knkski Have you tested this change on that version ?

[knkski]

I've only tested as far back as 1.21. It does look like mutating/validating webhooks in 1.18 have a failurePolicy field that allows for Fail or Ignore, so I'm not sure offhand why it wouldn't work. The docs say that the field defaults to Fail, which is what this change does anyways, so I can try just removing the field altogether.

As far as which version to release this in, no strong preference. Whichever works best for you.

[aws-kf-ci-bot]

@knkski: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
kubeflow-katib-presubmit	68e2761	link	/test kubeflow-katib-presubmit

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[tenzen-y]

Hi, @knkski !
I think that we need to specify Ignore in mutator.pod.katib.kubeflow.org since the katib-cert-generator job generates a Pod before the certificate is injected into MutatingWebhookConfiguration.

[knkski]

Apologies for the delay in responding, @DomFleischmann will be taking this over

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

We can close this in favour to PR: #2018