Add the steps to reboot the computes after update. #2587
Conversation
|
Thanks for the PR! ❤️ |
sathlan
force-pushed
the
update-reboot
branch
from
09bc6f0 to
0641dd5
Compare
|
Current tested with ping test running in the background and found not loss of connectivity. |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/4f12a60d8dc64dad92d4f7b5f5bec990 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 40m 14s |
sathlan
force-pushed
the
update-reboot
branch
from
0641dd5 to
a22e794
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/bf06493eaeea430c898bf25520dfdd04 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 44m 32s |
sathlan
force-pushed
the
update-reboot
branch
from
a22e794 to
4a5a0f2
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/caf50b2e762a4aaeb3326c9399dffd15 ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 21m 02s |
sathlan
force-pushed
the
update-reboot
branch
10 times, most recently
from
fdaa159 to
c056d3c
Compare
This sequence implements reboot of the compute nodes after the update. We have one instance created. If the hypervisor being rebooted has the instance that instance will be live-migrated to another hypervisor before the reboot and migrated back to that original hypervisor after the reboot. Some basic sanity checks are performed after the reboot and before the migration back to ensure that the necessary services are up and running. During the reboot we start two scripts. One monitors and log the reboot of the hypervisors. The other log where the instance is currently running. The log files can be found in `~/ci-framework-data/tests/update/` in `reboot_server_status.log` and `instance_placement.log` respectively. A note about node evacuation. We are still using node evaction from the nova cli. This command has not been ported to the openstack cli. There's a discussion about it [on launchpad](https://bugs.launchpad.net/python-openstackclient/+bug/2055552). The official documentation mention only the live-migration path, but as we also use the live-migration in the test sequence that part is covered. We still expect customer to use the nova cli as it's way more user friendly and is still currently working. Closes: https://issues.redhat.com/browse/OSPRH-8937
sathlan
force-pushed
the
update-reboot
branch
from
c056d3c to
85e4367
Compare
sathlan
added
enhancement
| PATH: "{{ cifmw_path | default(ansible_env.PATH) }}" | ||
| ansible.builtin.command: >- | ||
| {{ cifmw_update_oc_cmd_prefix }} | ||
| get openstackdataplanedeployment {{ cifmw_reboot_dep_name }} |
There was a problem hiding this comment.
| get openstackdataplanedeployment {{ cifmw_reboot_dep_name }} | |
| wait openstackdataplanedeployment {{ cifmw_reboot_dep_name }} | |
| --for=condition=ready | |
| --timeout={{ cifmw_update_timeout_reboot }}m |
With oc wait ansible log is more readable, as there no retires logged to output.
| edpm_reboot_strategy: force | ||
| ansibleLimit: {{ cifmw_update_hypervisor_short_name }} | ||
|
|
||
| - name: Apply the OpenStackDataPlaneDeployment CR to trigger a reboot |
There was a problem hiding this comment.
| - name: Apply the OpenStackDataPlaneDeployment CR to trigger a reboot | |
| - name: Create the OpenStackDataPlaneDeployment CR to trigger a reboot |
This sequence implements reboot of the compute nodes after the update.
We have one instance created. If the hypervisor being rebooted has
the instance that instance will be live-migrated to another hypervisor
before the reboot and migrated back to that original hypervisor after
the reboot.
Some basic sanity checks are performed after the reboot and before the
migration back to ensure that the necessary services are up and
running.
During the reboot we start two scripts. One monitors and log the
reboot of the hypervisors. The other log where the instance is
currently running. The log files can be found in
~/ci-framework-data/tests/update/inreboot_server_status.logandinstance_placement.logrespectively.A note about node evacuation. We are still using node evaction from
the nova cli. This command has not been ported to the openstack
cli. There's a discussion about it on launchpad.
The official documentation mention only the live-migration path, but
as we also use the live-migration in the test sequence that part is
covered. We still expect customer to use the nova cli as it's way
more user friendly and is still currently working.
Closes: https://issues.redhat.com/browse/OSPRH-8937