Getting LUKS & LVM to work with centos 7 -> rocky 8 in-place upgrade...

[verdverm]

Hi,

Still trying to get our inplace upgrade working. The issue we are facing is that leapp includes the systemd kernel module, which does not seem to play nicely with LUKS. Is that systemd kernel module needed for the initramfs-upgrade.img to work? Or maybe another module that depends on it?

thanks

[pirat89]

Hi, considering systemd kernel module as one of standard code modules that should be always present inside initramfs - and that's why you see it, it's put inside automatically. I would be surprised to see any initramfs without that module or its parts inside any initramfs so far. systemd module should not make any problems with LUKS at all.

Note that IPU with LUKS encrypted devices is not possible - and that's why it is inhibited. Currently we plan support LUKS2 for IPU 8 -> 9 only, when TPM2 token is used:

• Add LUKS support via Clevis TPM 2 token #1200

[verdverm]

Thanks, still trying to understand why it is not possible.

My understanding is that it was a blocker because it was not supported, instead of impossible. There are cases out there of people successfully upgrading with LUKS

[pirat89]

Well..
It's not supported because we are aware we do not have reliable solution at all. Well, that it could work for some particular setups, does not mean it will work always - especially when nobody is testing such a scenario. Yes, you are right that if you are ok to type password manually - and you have access to serial/physical console, then in some cases you are able to upgrade. No idea when yes, when no. I tried that after years just now on my my testing VM - most simple storage setup with luks (just 1 LVM LV containing everything but /boot; having everything local, no MP, etc..., using LUKS2), and removing the inhibitor I can do it.

But that works now. Years ago I have not been able to successfully upgrade even such a system. I have no idea what is different. Is it difference between RHEL 7.7 & 7.9? Is it 8.0 vs 8.10? Is it in our code? No idea honestly. And in the same time I am aware of a relatively recent customer case where someone tried the same thing (drop the actor checking storage encryption), and it could not boot to the upgrade environment due to locked encrypted storage, which has not been unlocked, and there was not any prompt asking for the password, as far as I remember from the investigation.

So I think I do not have answer for you when it's possible and when it doesn't when speaking about IPU 7 -> 8. Well, we could say that everything is possible, if you tweak it right to fit exactly your system. But then I do not have idea what everything you should do so you resolve all possible problems. It can be still completely unrelated issue around the storage initialisation (you can find couple of commits and experimental PR around that topic), but as far as you have encrypted storage, it can be also it.

Listing some known issues around storage if some of that take your attention:

• LVM + multipath? (missing filters in the upgrade initramfs)
• SAN over FC? LVM with more higher number LVs? (no enough time to initialise HW)
• SW RAID? - something is ok, something not. We already fixed some bugs, but no idea what everything could be broken.

Regarding encrypted storage, our plan is to focus on IPU 8 -> 9, starting to allow upgrades setups we consider reliable for the upgrade process.

[verdverm]

Thanks for the lengthy reply, I'm using a pretty vanilla (and older) centos 7 under qemu for testing luks

Our current hypothesis re: LUKS is that there is a difference between USB (where we keep the key) and on the RAID disks (SCSI in qemu testing). We are looking to temporarily move the key to onboard disks during the upgrade, maybe embed it into the initramfs and adjust references as needed

[verdverm]

Is the out-of-order logs here indicative that the disks are being discovered in the wrong order?

• sda is the encrypted disk
• sdb is the usb with luks key

[pirat89]

hmmm..seems like that. it's a question whether the system knows it should take a look info sdb at all for the key. no idea honestly 🤷 I am not familiar with the expected sequence of actions in this case (should it be hit on the first try or should it deal with the problem in a loop? I dunno.). It's also possible you would need more stuff in initramfs to deal with it.

@danzatt: possibly you would be interested about that. any idea? (it's an experiment, trying to upgrade with encrypted storage - without changes introduced in your PR I expect).

[verdverm]

One workaround I'm about to test is whether we can embed the key into the initramfs, adjust the lookup paths, and things work out.

I was able to get the key embed working by modifying the dracut call in the actor bash script to use the --include flag

Will keep he thread updated on how this works out

[verdverm]

We've been able to get past the luks issue by doing the following

• copy the decryption key(s) into the initramfs (see feature request: support custom content inside upgrade initramfs #1247)
• update the grub lines to reference the keys in the initramfs without using UUID mappings

We are now hitting issues with partition mounts & LVM, I'll follow up with another discussion, as it seems it may be helpful information for the leapp team and others. In brief

• /var is on a separate partition from /
• leapp was not detecting or mounting this like /usr
• we modified and added a /etc/fstab to the initramfs overlay we are using
• mostly works, but hitting another unknown issue now

Hangs at:

[verdverm]

Short update, we've been successful in our endeavor, will hopefully have time next week for the full write up