Error adding container to network "ovn-kubernetes": CNI request failed with status 400 #853

ksingh7 · 2021-09-03T21:26:03Z

ksingh7
Sep 3, 2021

Describe the bug

I am trying to install OKD on AWS, with 0 worker nodes and 1 Master Node. The OKD installer repeatedly fails

INFO Cluster operator kube-controller-manager Available is False with StaticPods_ZeroNodesActive: StaticPodsAvailable: 0 nodes are active; 1 nodes are at revision 0; 0 nodes have achieved new revision 3
ERROR Cluster operator kube-scheduler Degraded is True with InstallerPodContainerWaiting_ContainerCreating::InstallerPodNetworking_FailedCreatePodSandBox: InstallerPodContainerWaitingDegraded: Pod "installer-7-ip-10-0-136-203.ap-south-1.compute.internal" on node "ip-10-0-136-203.ap-south-1.compute.internal" container "installer" is waiting since 2021-09-03 20:45:54 +0000 UTC because ContainerCreating
ERROR InstallerPodNetworkingDegraded: Pod "installer-7-ip-10-0-136-203.ap-south-1.compute.internal" on node "ip-10-0-136-203.ap-south-1.compute.internal" observed degraded networking: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-7-ip-10-0-136-203.ap-south-1.compute.internal_openshift-kube-scheduler_8644b2c8-138f-46af-a507-135c7bb53bd7_0(de190dc1ab0cf13423a62dbed70f7a132adcc14c12fd77c076f00365ead1cc56): error adding pod openshift-kube-scheduler_installer-7-ip-10-0-136-203.ap-south-1.compute.internal to CNI network "multus-cni-network": [openshift-kube-scheduler/installer-7-ip-10-0-136-203.ap-south-1.compute.internal:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-kube-scheduler/installer-7-ip-10-0-136-203.ap-south-1.compute.internal de190dc1ab0cf13423a62dbed70f7a132adcc14c12fd77c076f00365ead1cc56] [openshift-kube-scheduler/installer-7-ip-10-0-136-203.ap-south-1.compute.internal de190dc1ab0cf13423a62dbed70f7a132adcc14c12fd77c076f00365ead1cc56] failed to get pod annotation: timed out waiting for annotations
ERROR InstallerPodNetworkingDegraded: '

Version

IPI on AWS
version : openshift-install-mac-4.8.0-0.okd-2021-09-03-092402

How reproducible
Tried twice but failed in both attempts, so 100% reproducible

Log bundle

Unfortunately the must-gather could not be run due to this issue

INFO Pulling debug logs from the bootstrap machine
FATAL failed to create SSH client: failed to use pre-existing agent, make sure the appropriate keys exist in the agent for authentication: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Hence sharing full install logs

INFO Credentials loaded from the "default" profile in file "/Users/karasing/.aws/credentials"
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings
INFO Consuming Install Config from target directory
INFO Creating infrastructure resources...
INFO Waiting up to 20m0s for the Kubernetes API at https://api.okd.scogo.in:6443...
INFO API v1.21.1-1398+98073871f173ba-dirty up
INFO Waiting up to 30m0s for bootstrapping to complete...
ERROR Cluster operator authentication Degraded is True with APIServerDeployment_PreconditionNotFulfilled::IngressStateEndpoints_MissingSubsets::OAuthAPIServerConfigObservation_Error::OAuthServerConfigObservation_Error::OAuthServerServiceEndpointAccessibleController_SyncError::OAuthServerServiceEndpointsEndpointAccessibleController_SyncError::RouterCerts_NoRouterCertSecret::WebhookAuthenticatorController_SyncError: APIServerDeploymentDegraded: waiting for observed configuration to have mandatory apiServerArguments.etcd-servers
ERROR APIServerDeploymentDegraded:
ERROR IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server
ERROR OAuthAPIServerConfigObservationDegraded: configmaps openshift-etcd/etcd-endpoints: no etcd endpoint addresses found
ERROR OAuthServerConfigObservationDegraded: secret "v4-0-config-system-router-certs" not found
ERROR OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.221.113:443/healthz": dial tcp 172.30.221.113:443: connect: connection refused
ERROR OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: oauth service endpoints are not ready
ERROR RouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret "v4-0-config-system-router-certs" not found
ERROR WebhookAuthenticatorControllerDegraded: failed to read service-ca crt bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory
INFO Cluster operator authentication Available is False with APIServerDeployment_PreconditionNotFulfilled::APIServices_PreconditionNotReady::OAuthServerServiceEndpointAccessibleController_EndpointUnavailable::OAuthServerServiceEndpointsEndpointAccessibleController_ResourceNotFound::ReadyIngressNodes_NoReadyIngressNodes: APIServicesAvailable: PreconditionNotReady
INFO OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.221.113:443/healthz": dial tcp 172.30.221.113:443: connect: connection refused
INFO OAuthServerServiceEndpointsEndpointAccessibleControllerAvailable: endpoints "oauth-openshift" not found
INFO ReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 1 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).
INFO Cluster operator baremetal Available is False with :
INFO Cluster operator baremetal Disabled is False with :
ERROR Cluster operator cluster-autoscaler Degraded is True with MissingDependency: machine-api not ready
INFO Cluster operator csi-snapshot-controller Progressing is True with _Deploying: Progressing: Waiting for Deployment to deploy csi-snapshot-controller pods
INFO Cluster operator csi-snapshot-controller Available is False with CSISnapshotWebhookController_Deploying::_Deploying: Available: Waiting for Deployment to deploy csi-snapshot-controller pods
INFO CSISnapshotWebhookControllerAvailable: Waiting for a validating webhook Deployment pod to start
INFO Cluster operator dns Available is False with DNSUnavailable: DNS "default" is unavailable.
INFO Cluster operator dns Progressing is True with DNSReportsProgressingIsTrue: DNS "default" reports Progressing=True: "Have 0 available DNS pods, want 1.\nHave 0 available node-resolver pods, want 1."
ERROR Cluster operator dns Degraded is True with DNSDegraded: DNS default is degraded
ERROR Cluster operator etcd Degraded is True with BootstrapTeardown_Error::EtcdMembersController_ErrorUpdatingReportEtcdMembers::InstallerController_Error: BootstrapTeardownDegraded: node lister not synced
ERROR EtcdMembersControllerDegraded: node lister not synced
ERROR InstallerControllerDegraded: missing required resources: [configmaps: etcd-peer-client-ca,etcd-serving-ca, configmaps: etcd-metrics-proxy-client-ca-2,etcd-pod-2,etcd-serving-ca-2]
INFO Cluster operator etcd Progressing is True with NodeInstaller: NodeInstallerProgressing: 1 nodes are at revision 0; 0 nodes have achieved new revision 2
INFO Cluster operator etcd Available is False with StaticPods_ZeroNodesActive: StaticPodsAvailable: 0 nodes are active; 1 nodes are at revision 0; 0 nodes have achieved new revision 2
INFO Cluster operator ingress Available is Unknown with IngressDoesNotHaveAvailableCondition: The "default" ingress controller is not reporting an Available status condition.
INFO Cluster operator ingress Progressing is True with Reconciling: Not all ingress controllers are available.
ERROR Cluster operator ingress Degraded is Unknown with IngressDoesNotHaveDegradedCondition: The "default" ingress controller is not reporting a Degraded status condition.
ERROR Cluster operator kube-apiserver Degraded is True with InstallerController_Error::StaticPods_Error: InstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,check-endpoints-kubeconfig,client-ca,control-plane-node-kubeconfig, secrets: service-network-serving-certkey, configmaps: bound-sa-token-signing-certs-2,config-2,etcd-serving-ca-2,kube-apiserver-audit-policies-2,kube-apiserver-cert-syncer-kubeconfig-2,kube-apiserver-pod-2,kubelet-serving-ca-2,sa-token-signing-certs-2, secrets: kubelet-client-2]
ERROR StaticPodsDegraded: pods "kube-apiserver-ip-10-0-136-203.ap-south-1.compute.internal" not found
INFO Cluster operator kube-apiserver Progressing is True with NodeInstaller: NodeInstallerProgressing: 1 nodes are at revision 0; 0 nodes have achieved new revision 2
INFO Cluster operator kube-apiserver Available is False with StaticPods_ZeroNodesActive: StaticPodsAvailable: 0 nodes are active; 1 nodes are at revision 0; 0 nodes have achieved new revision 2
ERROR Cluster operator kube-controller-manager Degraded is True with StaticPods_Error: StaticPodsDegraded: pod/kube-controller-manager-ip-10-0-136-203.ap-south-1.compute.internal container "cluster-policy-controller" is waiting: ContainerCreating:
ERROR StaticPodsDegraded: pod/kube-controller-manager-ip-10-0-136-203.ap-south-1.compute.internal container "kube-controller-manager" is waiting: ContainerCreating:
ERROR StaticPodsDegraded: pod/kube-controller-manager-ip-10-0-136-203.ap-south-1.compute.internal container "kube-controller-manager-cert-syncer" is waiting: ContainerCreating:
ERROR StaticPodsDegraded: pod/kube-controller-manager-ip-10-0-136-203.ap-south-1.compute.internal container "kube-controller-manager-recovery-controller" is waiting: ContainerCreating:
INFO Cluster operator kube-controller-manager Progressing is True with NodeInstaller: NodeInstallerProgressing: 1 nodes are at revision 0; 0 nodes have achieved new revision 3
INFO Cluster operator kube-controller-manager Available is False with StaticPods_ZeroNodesActive: StaticPodsAvailable: 0 nodes are active; 1 nodes are at revision 0; 0 nodes have achieved new revision 3
ERROR Cluster operator kube-scheduler Degraded is True with InstallerPodContainerWaiting_ContainerCreating::InstallerPodNetworking_FailedCreatePodSandBox: InstallerPodContainerWaitingDegraded: Pod "installer-7-ip-10-0-136-203.ap-south-1.compute.internal" on node "ip-10-0-136-203.ap-south-1.compute.internal" container "installer" is waiting since 2021-09-03 20:45:54 +0000 UTC because ContainerCreating
ERROR InstallerPodNetworkingDegraded: Pod "installer-7-ip-10-0-136-203.ap-south-1.compute.internal" on node "ip-10-0-136-203.ap-south-1.compute.internal" observed degraded networking: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-7-ip-10-0-136-203.ap-south-1.compute.internal_openshift-kube-scheduler_8644b2c8-138f-46af-a507-135c7bb53bd7_0(de190dc1ab0cf13423a62dbed70f7a132adcc14c12fd77c076f00365ead1cc56): error adding pod openshift-kube-scheduler_installer-7-ip-10-0-136-203.ap-south-1.compute.internal to CNI network "multus-cni-network": [openshift-kube-scheduler/installer-7-ip-10-0-136-203.ap-south-1.compute.internal:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-kube-scheduler/installer-7-ip-10-0-136-203.ap-south-1.compute.internal de190dc1ab0cf13423a62dbed70f7a132adcc14c12fd77c076f00365ead1cc56] [openshift-kube-scheduler/installer-7-ip-10-0-136-203.ap-south-1.compute.internal de190dc1ab0cf13423a62dbed70f7a132adcc14c12fd77c076f00365ead1cc56] failed to get pod annotation: timed out waiting for annotations
ERROR InstallerPodNetworkingDegraded: '
INFO Cluster operator kube-scheduler Progressing is True with NodeInstaller: NodeInstallerProgressing: 1 nodes are at revision 0; 0 nodes have achieved new revision 7
INFO Cluster operator kube-scheduler Available is False with StaticPods_ZeroNodesActive: StaticPodsAvailable: 0 nodes are active; 1 nodes are at revision 0; 0 nodes have achieved new revision 7
INFO Cluster operator kube-storage-version-migrator Available is False with _NoMigratorPod: Available: deployment/migrator.openshift-kube-storage-version-migrator: no replicas are available
INFO Cluster operator machine-api Progressing is True with SyncingResources: Progressing towards operator: 4.8.0-0.okd-2021-09-03-092402
ERROR Cluster operator machine-api Degraded is True with SyncingFailed: Failed when progressing towards operator: 4.8.0-0.okd-2021-09-03-092402 because deployment machine-api-controllers is not ready. status: (replicas: 1, updated: 1, ready: 0, unavailable: 1)
INFO Cluster operator machine-api Available is False with Initializing: Operator is initializing
INFO Cluster operator machine-config Progressing is True with : Working towards 4.8.0-0.okd-2021-09-03-092402
INFO Cluster operator monitoring Available is Unknown with :
INFO Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack.
ERROR Cluster operator monitoring Degraded is Unknown with :
INFO Cluster operator network ManagementStateDegraded is False with :
ERROR Cluster operator network Degraded is True with RolloutHung: DaemonSet "openshift-ovn-kubernetes/ovnkube-master" rollout is not making progress - last change 2021-09-03T20:44:15Z
INFO Cluster operator network Progressing is True with Deploying: DaemonSet "openshift-multus/multus-admission-controller" is waiting for other operators to become ready
INFO DaemonSet "openshift-ovn-kubernetes/ovnkube-master" is not available (awaiting 1 nodes)
INFO DaemonSet "openshift-network-diagnostics/network-check-target" is waiting for other operators to become ready
INFO Deployment "openshift-network-diagnostics/network-check-source" is waiting for other operators to become ready
INFO Cluster operator node-tuning Available is False with TunedUnavailable: DaemonSet "tuned" has no available Pod(s)
ERROR Cluster operator openshift-apiserver Degraded is True with APIServerDeployment_PreconditionNotFulfilled::ConfigObservation_Error: APIServerDeploymentDegraded: waiting for observed configuration to have mandatory StorageConfig.URLs
ERROR APIServerDeploymentDegraded:
ERROR ConfigObservationDegraded: configmaps openshift-etcd/etcd-endpoints: no etcd endpoint addresses found
INFO Cluster operator openshift-apiserver Available is False with APIServerDeployment_PreconditionNotFulfilled::APIServices_PreconditionNotReady: APIServicesAvailable: PreconditionNotReady
INFO Cluster operator openshift-controller-manager Progressing is True with _DesiredStateNotYetAchieved: Progressing: daemonset/controller-manager: number available is 0, desired number available > 1
INFO Cluster operator openshift-controller-manager Available is False with _NoPodsAvailable: Available: no daemon pods available on any node.
INFO Cluster operator operator-lifecycle-manager-catalog Progressing is True with : Deployed 0.17.0
INFO Cluster operator operator-lifecycle-manager-packageserver Available is False with :
INFO Cluster operator operator-lifecycle-manager-packageserver Progressing is True with : Working toward 0.17.0
INFO Cluster operator storage Progressing is True with AWSEBSCSIDriverOperatorCR_AWSEBSDriverControllerServiceController_Deploying: AWSEBSCSIDriverOperatorCRProgressing: AWSEBSDriverControllerServiceControllerProgressing: Waiting for Deployment to deploy pods
INFO Cluster operator storage Available is False with AWSEBSCSIDriverOperatorCR_AWSEBSDriverControllerServiceController_Deploying: AWSEBSCSIDriverOperatorCRAvailable: AWSEBSDriverControllerServiceControllerAvailable: Waiting for Deployment to deploy the CSI Controller Service
INFO Pulling debug logs from the bootstrap machine
ERROR Attempted to gather debug logs after installation failure: failed to create SSH client: failed to use pre-existing agent, make sure the appropriate keys exist in the agent for authentication: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
ERROR Bootstrap failed to complete: timed out waiting for the condition
ERROR Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane.
ERROR Attempted to analyze the debug logs after installation failure: could not open the gather bundle: open : no such file or directory
FATAL Bootstrap failed to complete

ksingh7 · 2021-09-03T21:26:50Z

ksingh7
Sep 3, 2021
Author

Sharing install-config.yaml

apiVersion: v1
baseDomain: scogo.in
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    aws:
      region: ap-south-1
      type: t3a.xlarge
      zones:
      - ap-south-1b
  replicas: 0
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    aws:
      region: ap-south-1
      type: m5a.2xlarge
      zones:
      - ap-south-1a
  replicas: 1
metadata:
  name: okd
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  aws:
    region: ap-south-1
    userTags:
      cluster: okd
publish: External

0 replies

ksingh7 · 2021-09-03T21:27:57Z

ksingh7
Sep 3, 2021
Author

BTW do you see any problem with the SSHKey entry in install-config.yaml, i am not sure if thats the reason why i was unable to run must-gather

sshKey: |
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmOwKKh4JQ8Swi9QDSq5qFIBV3QvxyMqCBbEwB5vg0pUH7FOI2K8uk5r3kgY71ICrZYkl7Ge+nih45lrcTGRVJ97NiCezYFalNo5wEhLeFxxIYlEuDVasdfasdqqwe2323Ax2YDBOeNBqNLR7/QBvdnQK/+SYLzj4t/3eUePazb8iJnn+zfYX+exQ2+/72RJzuN96lnJyMr+7OQ3p2zw7RTAf5eTbDB6BLp5J+zCnUlQp3kEZuHTqxgOVRsC7UryWVap/cXbavNicALlFhaoU0gCZgM8qypl2ZQ5UaTZOcqvJLvJi8RV85UqasJnT39TIPcT/462egN karasing@karasing-OSX

0 replies

ksingh7 · 2021-09-04T18:30:52Z

ksingh7
Sep 4, 2021
Author

Today i changed the install-config.yaml file and set 3 replicas for masters and 1 replica for worker and it failed again but this time witha different error

INFO Credentials loaded from the "default" profile in file "/Users/karasing/.aws/credentials"
INFO Consuming Install Config from target directory
INFO Creating infrastructure resources...
INFO Waiting up to 20m0s for the Kubernetes API at https://api.okd.scogo.in:6443...
INFO API v1.21.1-1398+98073871f173ba-dirty up
INFO Waiting up to 30m0s for bootstrapping to complete...
INFO Destroying the bootstrap resources...
INFO Waiting up to 40m0s for the cluster at https://api.okd.scogo.in:6443 to initialize...
INFO Cluster operator baremetal Disabled is True with UnsupportedPlatform: Nothing to do on this Platform
INFO Cluster operator insights Disabled is False with AsExpected:
INFO Cluster operator machine-config Progressing is True with : Working towards 4.8.0-0.okd-2021-09-03-092402
ERROR Cluster operator machine-config Degraded is True with RequiredPoolsFailed: Unable to apply 4.8.0-0.okd-2021-09-03-092402: timed out waiting for the condition during syncRequiredMachineConfigPools: error pool master is not ready, retrying. Status: (pool degraded: true total: 3, ready 0, updated: 0, unavailable: 3)
INFO Cluster operator machine-config Available is False with : Cluster not available for 4.8.0-0.okd-2021-09-03-092402
INFO Cluster operator network ManagementStateDegraded is False with :
INFO Cluster operator openshift-controller-manager Progressing is True with _DesiredStateNotYetAchieved: Progressing: daemonset/controller-manager: updated number scheduled is 2, desired number scheduled is 3
ERROR Cluster initialization failed because one or more operators are not functioning properly.
ERROR The cluster should be accessible for troubleshooting as detailed in the documentation linked below,
ERROR https://docs.openshift.com/container-platform/latest/support/troubleshooting/troubleshooting-installations.html
ERROR The 'wait-for install-complete' subcommand can then be used to continue the installation
FATAL failed to initialize the cluster: Cluster operator machine-config is not available

0 replies

JaimeMagiera · 2021-09-04T18:44:59Z

JaimeMagiera
Sep 4, 2021
Maintainer

Please see the link below for acceptable values for control and compute nodes. In particular, you need at least 3 control and 2 compute.

https://docs.okd.io/latest/installing/installing_aws/installing-aws-customizations.html#installation-configuration-parameters_installing-aws-customizations

0 replies

ksingh7 · 2021-09-04T19:39:17Z

ksingh7
Sep 4, 2021
Author

Thanks @JaimeMagiera for the pointer, however the OKD FAQ mentions that we can run a single node cluster

https://github.com/openshift/okd/blob/master/FAQ.md#can-i-run-a-single-node-cluster

7 replies

ksingh7 Sep 5, 2021
Author

Hey, @JaimeMagiera really appreciate your help on this.

ksingh7 Sep 5, 2021
Author

@JaimeMagiera @vladikr I read somewhere on the internet that

The only problem that single node OKD cluster are working until version 4.5.
Everything above 4.5 doesn’t work anymore and the installer requires 3x masters nodes at least.

Is this true ?

ksingh7 Sep 5, 2021
Author

Looks like this theory is correct, i tried installing single node OKD 4.7 (several different releases) without any success. Finally, i tried OKD 4.5 and in one attempt my single node cluster got deployed like a charm

INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/Users/karasing/Downloads/okd-setup/openshift-install-mac-4.5.0-0.okd-2020-10-15-235428/scogo-okd-1/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.okd.scogo.in
INFO Login to the console with user: "kubeadmin", and password: "8V7LD-wQmGv-SWkMF-QWESX"
DEBUG Time elapsed per stage:
DEBUG     Infrastructure: 6m21s
DEBUG Bootstrap Complete: 11m59s
DEBUG                API: 2m26s
DEBUG  Bootstrap Destroy: 52s
DEBUG  Cluster Operators: 9m35s
INFO Time elapsed: 28m59s

JaimeMagiera Sep 5, 2021
Maintainer

It doesn't seem to be possible and until we can clearly prove that it is, I've submitted a pull request to update the FAQ.

#858

ksingh7 Sep 6, 2021
Author

@JaimeMagiera ACK

Not sure but if possible for you and you have time, can you give a quick test with OKD 4.7 and beyond with 1 master and 0 worker nodes on IPI AWS.

Again thanks for helping on this.

vrutkovs · 2021-09-06T07:54:31Z

vrutkovs
Sep 6, 2021
Maintainer

INFO Pulling debug logs from the bootstrap machine
ERROR Attempted to gather debug logs after installation failure: failed to create SSH client: failed to use pre-existing agent, make sure the appropriate keys exist in the agent for authentication: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
ERROR Bootstrap failed to complete: timed out waiting for the condition

Bootstrap failed to pivot into OKD content? Is it running and can it be ssh'ed into?

1 reply

ksingh7 Sep 6, 2021
Author

@vrutkovs, unfortunately, I do not have that env now, i treated it off and deployed single node OKD 4.5 (which worked)

However, @JaimeMagiera graciously tested single node OKD 4.7 that failed for him. you can find the log bundle below

JaimeMagiera · 2021-09-06T16:31:53Z

JaimeMagiera
Sep 6, 2021
Maintainer

Version: 4.7.0-0.okd-2021-08-22-163618
Provider: AWS (IPI)
Control: 1
Compute: 0
Result: Failure

log-bundle-20210906121344.tar.gz

1 reply

ksingh7 Sep 6, 2021
Author

@JaimeMagiera Thanks for testing this, really appreciate this.

@vrutkovs are we missing something on single node OKD cluster successful deployment post OKD 4.5 release?

ksingh7 · 2021-09-06T19:06:25Z

ksingh7
Sep 6, 2021
Author

And just for the record, i read in this blog post comment about single-node clusters no longer works with versions > 4.5

https://techbloc.net/archives/4672#comment-169663

0 replies

JaimeMagiera · 2021-09-07T00:16:32Z

JaimeMagiera
Sep 7, 2021
Maintainer

Version: 4.8.0-0.okd-2021-09-06-195513
Provider: AWS (IPI)
Control: 1
Compute: 0
Result: Success

3 replies

ksingh7 Sep 19, 2021
Author

@JaimeMagiera can you share your install-config.yaml that was successful in this test (by hiding critical info) , just wanted to verify if i am not doing anything wrong, as i am still unable to get single node AWS IP OKD 4.8 (my rest results are below)

ksingh7 Sep 19, 2021
Author

@JaimeMagiera I tried pulling binaries for the exact same version that worked for you but failed

oc adm release extract --tools registry.ci.openshift.org/origin/release:4.8.0-0.okd-2021-09-06-195513
error: image "registry.ci.openshift.org/origin/release:4.8.0-0.okd-2021-09-06-195513" not found: manifest unknown: manifest unknown

JaimeMagiera Sep 19, 2021
Maintainer

That was a nightly build, as opposed to a release build. Nightly builds are pruned after 72 hours. So, you'd want to try one of the more recent 4.8 builds.

JaimeMagiera · 2021-09-07T01:13:12Z

JaimeMagiera
Sep 7, 2021
Maintainer

Version: 4.6.0-0.okd-2021-09-02-135015
Provider: AWS (IPI)
Control: 1
Compute: 0
Result: Failure

log-bundle-20210906211011.tar.gz

0 replies

ksingh7 · 2021-09-19T07:53:50Z

ksingh7
Sep 19, 2021
Author

@JaimeMagiera I tried to install single node OKD 4.8.0-0.okd-2021-09-18-130143 version but it failed again.

Provider: AWS (IPI)
Control: 1
Compute: 0
Result: Failure

INFO Cluster operator machine-config Progressing is True with : Working towards 4.8.0-0.okd-2021-09-18-130143
ERROR Cluster operator machine-config Degraded is True with RequiredPoolsFailed: Unable to apply 4.8.0-0.okd-2021-09-18-130143: timed out waiting for the condition during syncRequiredMachineConfigPools: error pool master is not ready, retrying. Status: (pool degraded: true total: 1, ready 0, updated: 0, unavailable: 1)
INFO Cluster operator machine-config Available is False with : Cluster not available for 4.8.0-0.okd-2021-09-18-130143
INFO Cluster operator network ManagementStateDegraded is False with :
ERROR Cluster initialization failed because one or more operators are not functioning properly.
ERROR The cluster should be accessible for troubleshooting as detailed in the documentation linked below,
ERROR https://docs.openshift.com/container-platform/latest/support/troubleshooting/troubleshooting-installations.html
ERROR The 'wait-for install-complete' subcommand can then be used to continue the installation
FATAL failed to initialize the cluster: Cluster operator machine-config is not available

3 replies

JaimeMagiera Sep 19, 2021
Maintainer

Let me give that build a shot and see what happens. Ultimately, I'll be setting up a CI pipeline for single-node cluster installs to more quickly find and address bugs.

JaimeMagiera Sep 19, 2021
Maintainer

I was able to build 4.8.0-0.okd-2021-09-18-130143 successfully. In fact, it's now in the process of updating to 4.8.0-0.okd-2021-09-18-201810.

Set the AWS profile

export AWS_PROFILE= my-profile

Generate config file

bin/openshift-install create install-config --dir=$(pwd)

Change the values for the replicas of control plane and compute

sed -i '0,/replicas: 3/ {0,/replicas: 3/ s/replicas: 3/replicas: 0/}' install-config.yaml
sed -i '0,/replicas: 3/ {0,/replicas: 3/ s/replicas: 3/replicas: 1/}' install-config.yaml

Run the install

bin/openshift-install create cluster --dir=$(pwd) --log-level=info

Win.

Are you changing anything else in your config file? Are you allowing the installer to create a new VPC or trying to re-use an existing one? Is your DNS correct?

ksingh7 Sep 20, 2021
Author

@JaimeMagiera Thanks again for sparing time on testing this, appreciated it.

Yes DNS is correct, 3 node cluster works well with this config.

Let me follow the above steps strictly and see how it looks like.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error adding container to network "ovn-kubernetes": CNI request failed with status 400 #853

{{title}}

Replies: 11 comments 15 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Error adding container to network "ovn-kubernetes": CNI request failed with status 400 #853

ksingh7 Sep 3, 2021

Replies: 11 comments · 15 replies

ksingh7 Sep 3, 2021 Author

ksingh7 Sep 3, 2021 Author

ksingh7 Sep 4, 2021 Author

JaimeMagiera Sep 4, 2021 Maintainer

ksingh7 Sep 4, 2021 Author

ksingh7 Sep 5, 2021 Author

ksingh7 Sep 5, 2021 Author

ksingh7 Sep 5, 2021 Author

JaimeMagiera Sep 5, 2021 Maintainer

ksingh7 Sep 6, 2021 Author

vrutkovs Sep 6, 2021 Maintainer

ksingh7 Sep 6, 2021 Author

JaimeMagiera Sep 6, 2021 Maintainer

ksingh7 Sep 6, 2021 Author

ksingh7 Sep 6, 2021 Author

JaimeMagiera Sep 7, 2021 Maintainer

ksingh7 Sep 19, 2021 Author

ksingh7 Sep 19, 2021 Author

JaimeMagiera Sep 19, 2021 Maintainer

JaimeMagiera Sep 7, 2021 Maintainer

ksingh7 Sep 19, 2021 Author

JaimeMagiera Sep 19, 2021 Maintainer

JaimeMagiera Sep 19, 2021 Maintainer

ksingh7 Sep 20, 2021 Author

ksingh7
Sep 3, 2021

Replies: 11 comments 15 replies

ksingh7
Sep 3, 2021
Author

ksingh7
Sep 3, 2021
Author

ksingh7
Sep 4, 2021
Author

JaimeMagiera
Sep 4, 2021
Maintainer

ksingh7
Sep 4, 2021
Author

ksingh7 Sep 5, 2021
Author

ksingh7 Sep 5, 2021
Author

ksingh7 Sep 5, 2021
Author

JaimeMagiera Sep 5, 2021
Maintainer

ksingh7 Sep 6, 2021
Author

vrutkovs
Sep 6, 2021
Maintainer

ksingh7 Sep 6, 2021
Author

JaimeMagiera
Sep 6, 2021
Maintainer

ksingh7 Sep 6, 2021
Author

ksingh7
Sep 6, 2021
Author

JaimeMagiera
Sep 7, 2021
Maintainer

ksingh7 Sep 19, 2021
Author

ksingh7 Sep 19, 2021
Author

JaimeMagiera Sep 19, 2021
Maintainer

JaimeMagiera
Sep 7, 2021
Maintainer

ksingh7
Sep 19, 2021
Author

JaimeMagiera Sep 19, 2021
Maintainer

JaimeMagiera Sep 19, 2021
Maintainer

ksingh7 Sep 20, 2021
Author