-
Notifications
You must be signed in to change notification settings - Fork 148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
IPU 9 -> 10: Initial preparation: upgrade paths and leapp packages def #1169
Conversation
Thank you for contributing to the Leapp project!Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build. If you need a different version of leapp from PR#42, use It is possible to schedule specific on-demand tests as well. Currently 2 test sets are supported,
[Deprecated] To launch on-demand regression testing public members of oamg organization can leave the following comment:
Please open ticket in case you experience technical problem with the CI. (RH internal only) Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra. |
/packit copr-build |
9c512d4
to
1e42b2b
Compare
/packit copr-build |
708e812
to
c8898da
Compare
Rebased to include new device deprecation data and PES events. |
40e2010
to
0e24ae2
Compare
8261889
to
dd8070f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Before it could be merged, we need to:
- we need to polish commits - there is number of commits that are basically dead - on top of sqash & fixup commits
- partially done. keeping the final round when tests are green again
- update repomap.json
- it needs to be rebased against
master
to resolve conflict
repos/system_upgrade/common/actors/checksaphana/libraries/checksaphana.py
Show resolved
Hide resolved
repos/system_upgrade/common/actors/opensshpermitrootlogincheck/actor.py
Outdated
Show resolved
Hide resolved
repos/system_upgrade/common/actors/removeobsoletegpgkeys/libraries/removeobsoleterpmgpgkeys.py
Outdated
Show resolved
Hide resolved
5b19dcd
to
ce4e56b
Compare
Introducing initial changes needed for IPU 9 -> 10 to prevent abouvious crashes and inhibitors: * Extend the list of supported source versions for the upgrade by el9 * Define upgrade paths for IPU 9 -> 10 * Add Python path for el 10 (py 3.12) * Introducing el9toel10 repository created .gitkeep files inside empty dirs NOTE: Enable upgrade from RHEL 9.4 & 9.5 to RHEL 10.0 for now. For the experiment purposes. These will not be supported for the upgrade at all, but to make testing and experimentations easier, allow it for now. TODO: * add RHEL 10 product certificates * add RHEL 10 GPG keys
@matejmatuska resolved the problem with distutils. keeping it in a separate commit for possible future investigations of bugs I introduced inside :-D (there is always bug) |
@pirat89 The code looks alright. I suggest adding a unit test(s) as there currently aren't any IIRC. (or I can add some) |
Calling it EOD. Some tests would be nice really, but as they are missing all the time, considering it nice to have in this situation. But it would be still better to have written at least TODO or something that they are actually missing. If you want, feel free to add them. Hmm...or maybe we could try to use AI :-P I've realized it's not bad when speaking about unit-tests. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use e.g. jq < repomap.json
to perform basic validation of json
Based on the bug fixed in the previuos commit, we are rather adding unit-tests for get_major_version() function in common libraries, to prevent possible issues in future there too.
Note that the data is artificial, meaning it's not generated through repomapping generator and therefore not verified by it against the source of truth. The assumption is that the repos will be similar to how they are on RHEL 9 and are basically copied (redundant channels and rhui repos are dropped).
They are not owned by any package and can dynamically grow to a huge amount of files causing hitting open files limit
The RPM DB has been moved from /var/lib/rpm to /usr/lib/sysimage/rpm in RHEL 10. Apply the change and create symlink in the original path to the new one as expected. Also rebuild the RPM DB to ensure it's compatible with the new RPM version. Previously the RPM DB was being rebuilt during IPU 8 -> 9. However after discussion with RPM SMEs it has been decided that this is actually very reasonable to do always. So applying for any upgrade path but IPU 7 -> 8 (let's do not change this one so much when it's kind of finished). Co-authored-by: Matej Matuska <[email protected]>
No keys will be obosoleted, however it's expected to define the list.
The `onerror` argument of shutil.rmtree is deprecated and replaced by `onexc` since Python 3.12. It accepts callback with the same arguments except for the last one which is now a subclass of `BaseException`. Our callback `libraries.utils.report_and_ignore_shutil_rmtree_error` is thus modified accordingly.
Originally the actor used distutils.version.LooseVersion to detect newest version of installed kernel package inside target userspace container. But the distutils python module has become deprecated and we should not use it anymore in Python 3.12+. However, we do not expect to see multiple versions of kernel present inside the target userspace container as the container is created during the IPU process from scratch (always). Considering the presence of multiple kernels to be sign of error, which could negatively affect also additional actions later. Updated the solution, raising an error if multiple kernels are detected inside the container. As we expect one kernel only, no need to compare versions of particular possible kernel packages. Note there are now just these cases when this could happen: * a third party package is required to be installed inside the container * an explicit requirement to install particular version of container is made by a custom actor
Required changes have been addressed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great job all! LGTM :)
@Rezney @matejmatuska great job guys!! |
## Packaging - .. names of packages, dependencies, changes in provided capabilities.... ## Upgrade handling ### Fixes - Add missing RHUI GCP config info for RHEL for SAP (oamg#1253) - Fix creation of the post upgrade report about changes in states of systemd services (oamg#1210) - Fix detection of valid sshd config with internal-sftp subsystem in Leapp (oamg#1212) - Fix evaluation of PES data (oamg#1194) - Fix failing "update-ca-trust" command caused by missing util-linux package (oamg#1169) - Fix handling of versions in RHUI configuration for ELS and SAP upgrades (oamg#1240) - Fix the parsing of the lscpu output (oamg#1184, oamg#1208) - Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (oamg#1178) - Fix upgrade on aarch64 via RHUI on AWS (oamg#1240) - Handle a false positive GPG check error when TargetUserSpaceInfo is missing (oamg#1269) - Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (oamg#1205) - Update the default kernel cmdline (oamg#1193, oamg#1216) - Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (oamg#1211) - Wait for the storage initialization when /usr is on separate file system - covering SAN (oamg#1218, oamg#1219) - [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (oamg#1243) - [IPU 7 -> 8] Fix detection of bootable device on RAID (oamg#1260) - [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (oamg#1270) ### Enhancements - [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (oamg#1245, oamg#1246) - Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (oamg#1199) - Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (oamg#1197) - Improve set_systemd_services_states logging (oamg#1213) - Include leapp command execution and defined leapp envars inside leapp.db - (oamg#1152) - Introduce experimental upgrades in 'live' mode for the testing (oamg#1248) - Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (oamg#1241) - Several minor improvements in messages printed in console output (oamg#1173, oamg#1214, oamg#1274) - Several minor improvements in report and error messages (oamg#1207, oamg#1217, oamg#1234, oamg#1235, oamg#1242) - Sort lists in dnf-plugin-data for easier overview (oamg#1231) - [IPU 7 -> 8] Allow upgrade of content from ELS repositories (oamg#1198) - [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (oamg#1206) - [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (oamg#1195) - [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (oamg#1249) - [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (oamg#1181) - [IPU 9 -> 10] Introduced number of changes to enable experimental IPU 9 -> 10 (oamg#1169) - [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (oamg#1268) - [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (oamg#1169, oamg#1273) ## Additional changes interesting for devels - drop unused `packager` field from gpg-signatures.json (oamg#1233) - [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible - [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo
## Packaging - Start building for EL 9 in the upstream repository on COPR (#1169) ## Upgrade handling ### Fixes - Add missing RHUI GCP config info for RHEL for SAP (#1253) - Fix creation of the post upgrade report about changes in states of systemd services (#1210) - Fix detection of valid sshd config with internal-sftp subsystem in Leapp (#1212) - Fix evaluation of PES data (#1194) - Fix failing "update-ca-trust" command caused by missing util-linux package (#1169) - Fix handling of versions in RHUI configuration for ELS and SAP upgrades (#1240) - Fix the parsing of the lscpu output (#1184, #1208) - Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (#1178) - Fix upgrade on aarch64 via RHUI on AWS (#1240) - Handle a false positive GPG check error when TargetUserSpaceInfo is missing (#1269) - Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (#1205) - Update the default kernel cmdline (#1193, #1216) - Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (#1211) - Wait for the storage initialization when /usr is on separate file system - covering SAN (#1218, #1219) - [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (#1243) - [IPU 7 -> 8] Fix detection of bootable device on RAID (#1260) - [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (#1270) ### Enhancements - [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (#1245, #1246) - Update leapp data files (#1280) - Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (#1199) - Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (#1197) - Improve set_systemd_services_states logging (#1213) - Include leapp command execution and defined leapp envars inside leapp.db - (#1152) - Introduce experimental upgrades in 'live' mode for the testing (#1248) - Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (#1241) - Several minor improvements in messages printed in console output (#1173, #1214, #1274) - Several minor improvements in report and error messages (#1207, #1217, #1234, #1235, #1242) - Sort lists in dnf-plugin-data for easier overview (#1231) - [IPU 7 -> 8] Allow upgrade of content from ELS repositories (#1198) - [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (#1206) - [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (#1195) - [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (#1249) - [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (#1181) - [IPU 9 -> 10] Introduced number of changes to enable IPU 9 -> 10 for testing (#1169) - [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (#1268) - [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (#1169, #1273) ## Additional changes interesting for devels - drop unused `packager` field from gpg-signatures.json (#1233) - [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible - [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo
## Packaging - Start building for EL 9 in the upstream repository on COPR (oamg#1169) ## Upgrade handling ### Fixes - Add missing RHUI GCP config info for RHEL for SAP (oamg#1253) - Fix creation of the post upgrade report about changes in states of systemd services (oamg#1210) - Fix detection of valid sshd config with internal-sftp subsystem in Leapp (oamg#1212) - Fix evaluation of PES data (oamg#1194) - Fix failing "update-ca-trust" command caused by missing util-linux package (oamg#1169) - Fix handling of versions in RHUI configuration for ELS and SAP upgrades (oamg#1240) - Fix the parsing of the lscpu output (oamg#1184, oamg#1208) - Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (oamg#1178) - Fix upgrade on aarch64 via RHUI on AWS (oamg#1240) - Handle a false positive GPG check error when TargetUserSpaceInfo is missing (oamg#1269) - Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (oamg#1205) - Update the default kernel cmdline (oamg#1193, oamg#1216) - Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (oamg#1211) - Wait for the storage initialization when /usr is on separate file system - covering SAN (oamg#1218, oamg#1219) - [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (oamg#1243) - [IPU 7 -> 8] Fix detection of bootable device on RAID (oamg#1260) - [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (oamg#1270) ### Enhancements - [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (oamg#1245, oamg#1246) - Update leapp data files (oamg#1280) - Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (oamg#1199) - Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (oamg#1197) - Improve set_systemd_services_states logging (oamg#1213) - Include leapp command execution and defined leapp envars inside leapp.db - (oamg#1152) - Introduce experimental upgrades in 'live' mode for the testing (oamg#1248) - Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (oamg#1241) - Several minor improvements in messages printed in console output (oamg#1173, oamg#1214, oamg#1274) - Several minor improvements in report and error messages (oamg#1207, oamg#1217, oamg#1234, oamg#1235, oamg#1242) - Sort lists in dnf-plugin-data for easier overview (oamg#1231) - [IPU 7 -> 8] Allow upgrade of content from ELS repositories (oamg#1198) - [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (oamg#1206) - [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (oamg#1195) - [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (oamg#1249) - [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (oamg#1181) - [IPU 9 -> 10] Introduced number of changes to enable IPU 9 -> 10 for testing (oamg#1169) - [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (oamg#1268) - [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (oamg#1169, oamg#1273) ## Additional changes interesting for devels - drop unused `packager` field from gpg-signatures.json (oamg#1233) - [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible - [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo (cherry picked from commit 03c257b)
## Packaging - Start building for EL 9 in the upstream repository on COPR (oamg#1169) ## Upgrade handling ### Fixes - Add missing RHUI GCP config info for RHEL for SAP (oamg#1253) - Fix creation of the post upgrade report about changes in states of systemd services (oamg#1210) - Fix detection of valid sshd config with internal-sftp subsystem in Leapp (oamg#1212) - Fix evaluation of PES data (oamg#1194) - Fix failing "update-ca-trust" command caused by missing util-linux package (oamg#1169) - Fix handling of versions in RHUI configuration for ELS and SAP upgrades (oamg#1240) - Fix the parsing of the lscpu output (oamg#1184, oamg#1208) - Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (oamg#1178) - Fix upgrade on aarch64 via RHUI on AWS (oamg#1240) - Handle a false positive GPG check error when TargetUserSpaceInfo is missing (oamg#1269) - Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (oamg#1205) - Update the default kernel cmdline (oamg#1193, oamg#1216) - Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (oamg#1211) - Wait for the storage initialization when /usr is on separate file system - covering SAN (oamg#1218, oamg#1219) - [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (oamg#1243) - [IPU 7 -> 8] Fix detection of bootable device on RAID (oamg#1260) - [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (oamg#1270) ### Enhancements - [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (oamg#1245, oamg#1246) - Update leapp data files (oamg#1280) - Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (oamg#1199) - Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (oamg#1197) - Improve set_systemd_services_states logging (oamg#1213) - Include leapp command execution and defined leapp envars inside leapp.db - (oamg#1152) - Introduce experimental upgrades in 'live' mode for the testing (oamg#1248) - Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (oamg#1241) - Several minor improvements in messages printed in console output (oamg#1173, oamg#1214, oamg#1274) - Several minor improvements in report and error messages (oamg#1207, oamg#1217, oamg#1234, oamg#1235, oamg#1242) - Sort lists in dnf-plugin-data for easier overview (oamg#1231) - [IPU 7 -> 8] Allow upgrade of content from ELS repositories (oamg#1198) - [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (oamg#1206) - [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (oamg#1195) - [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (oamg#1249) - [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (oamg#1181) - [IPU 9 -> 10] Introduced number of changes to enable IPU 9 -> 10 for testing (oamg#1169) - [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (oamg#1268) - [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (oamg#1169, oamg#1273) ## Additional changes interesting for devels - drop unused `packager` field from gpg-signatures.json (oamg#1233) - [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible - [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo (cherry picked from commit 03c257b)
Introducing initial changes needed for IPU 9 -> 10 to prevent abouvious crashes and inhibitors:
get_leapp_packages
andget_leapp_dep_packages
LEAPP_IPU_IN_PROGRESS
for IPU 9->10 which set originally just9to1
TODO (required bare minimum changes for the IPU 9 -> 10):