K8SPXC-1155: Fix stuck smart update after invalid configuration

[egegunes]

CHANGE DESCRIPTION

Problem:
If user applies a config which causes pods fail to schedule, they can't fix it by applying corrected configuration because operator rejects to start smart update if not all PXC pods are ready.

Solution:

• This PR removes the check to wait for all PXC pods be ready before smart update.
• If pods fail to schedule, operator retries to smart update for 2 hours by default. This is very long to fail smart updated. After these changes, if a pod is failed to schedule for 2 minutes operator will fail the smart update and return an error.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?
• Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PXC version?
• Does the change support oldest and newest supported Kubernetes version?

[JNKPercona]

Test name	Status
affinity-8-0	passed
auto-tuning-8-0	passed
cross-site-8-0	passed
demand-backup-cloud-8-0	passed
demand-backup-encrypted-with-tls-8-0	passed
demand-backup-8-0	passed
haproxy-5-7	passed
haproxy-8-0	passed
init-deploy-5-7	passed
init-deploy-8-0	passed
limits-8-0	passed
monitoring-2-0-8-0	passed
one-pod-5-7	passed
one-pod-8-0	passed
pitr-8-0	passed
pitr-gap-errors-8-0	passed
proxy-protocol-8-0	passed
proxysql-sidecar-res-limits-8-0	passed
pvc-resize-5-7	passed
pvc-resize-8-0	passed
recreate-8-0	passed
restore-to-encrypted-cluster-8-0	passed
scaling-proxysql-8-0	passed
scaling-8-0	passed
scheduled-backup-5-7	passed
scheduled-backup-8-0	passed
security-context-8-0	passed
smart-update1-8-0	passed
smart-update2-8-0	passed
storage-8-0	passed
tls-issue-cert-manager-ref-8-0	passed
tls-issue-cert-manager-8-0	passed
tls-issue-self-8-0	passed
upgrade-consistency-8-0	passed
upgrade-haproxy-5-7	passed
upgrade-haproxy-8-0	passed
upgrade-proxysql-5-7	passed
upgrade-proxysql-8-0	passed
users-5-7	passed
users-8-0	passed
validation-hook-8-0	passed
We run 41 out of 41

commit: ea892c3
image: perconalab/percona-xtradb-cluster-operator:PR-1829-ea892c39