consider verifying signatures in userspace

[cgwalters]

Migrating this from ostreedev/ostree#2879 (comment)

Today, our signature verification logic relies on the in-kernel fsverity signature handling.

In the primary original use case for fsverity (e.g. Android), signatures on the files are verified in userspace before they're processed. Now, a whole problem with using fsverity outside of Android is other Linux systems don't ship apps as single .zip files with a single trusted process launcher.

But, composefs is a way to sign and manage filesystem trees - and the fsverity maintainer is arguing that it makes more sense for us to do signature verification in userspace, instead of going through the Linux kernel's fsverity "automatic" flow using CONFIG_FS_VERITY_BUILTIN_SIGNATURES.

I think I lean in that direction too. At least, we should support external/userspace signatures and not require CONFIG_FS_VERITY_BUILTIN_SIGNATURES. Which I guess we basically do now because we could just document how to use whatever tools (e.g. openssl) in combination with calling FS_IOC_MEASURE_VERITY on the erofs to verify the signature before mounting.

(This topic relates to the question of how opinionated this project is, which relates to #125 )

Hmm. One tricky thing here is that if we say that the signed object is the fsverity digest (as we do now), that then does really commit us to fsverity for the erofs metadata file. But long term...it may actually make sense to cut the backing filesystem out of the flow for the erofs metadata (i.e. not use loopback files...)? In a non-loopback world, perhaps we actually use dm-verity for the erofs metadata? I guess nothing really stops us today actually from setting up dm-verity on the loopback and using its signature tooling...although that cuts strongly against the "block device is hidden" argument.

Well, anyways I guess the bottom line here is that in theory, we do support "external signatures" today. But we should document it. And then a debate is whether to keep the current signature code which the fsverity maintainer argues against.

[hsiangkao]

In the mid term, EROFS could have a built-in verity mechanism as well, because fsverity or dmverity are just a container format, it's not the part of EROFS on-disk format. It has pros and cons too.

[alexlarsson]

At the end of the day, composefs supports verifying a mount against a given digest, so any userspace code can on its own have a digest + signature of the digest, verity that and pass the verified digest to composefs.

So, technically we do support verifying signatures in userspace already. However, maybe we want to have some standardized way to do this with some code in libcomposefs backing it.

I guess one offshot of this is that we should perhaps limit how much we recommend the current fs-verity signatures. For example, we should probably rename LCFS_MOUNT_FLAGS_REQUIRE_SIGNATURE to make it explicit which kind of signature it refers to.

In a non-loopback world, perhaps we actually use dm-verity for the erofs metadata?

I don't understand this. When we are using loopback we could use either dm-verity of fs-verity, but if we move to drop loopback then we have no block device, so dm-verity is not usable (but fs-verity still is).

But again, maybe we should be more explicit about this too. Soo we can change expected_digest to expected_fsverity_digest, in case we later also have other digests.

[alexlarsson]

Anyway, I gotta say that I'm personally not competent enough to do a userspace implementation of signature validation.

In theory it is rather simple:

• Take the fs-verity digest, sign it with a private key, store signature in a file
• At mount time get verity digest of the image and the signature, verify these against some public key (in a file)
• Pass the verified digest to mount.composefs (or pass in the fd you got the verity digest from)

However, there are lots of open questions:

• Where do you find/store the key?
• How do you trust that someone didn't put in their own public key?
• Can we do trust chains? key aging? revocation?
• What file formats should it use?
• What algorithms should it use?

Basically, someone who knows this kind of stuff like the back of their hand need to do this.

[cgwalters]

At the end of the day, composefs supports verifying a mount against a given digest, so any userspace code can on its own have a digest + signature of the digest, verity that and pass the verified digest to composefs.

Right, so I think the first deliverable here is just to document this. I may tackle this after #150 merges.

But I do also lean towards having some standardized flow for this. Hmm...actually, since EROFS reserves the first 1k (right?) for empty space for BIOS MBR-related compat reasons (which is kind of silly that we're doing that even with cfs, but that's an aside) - we could stick a signature there?

When we are using loopback we could use either dm-verity of fs-verity, but if we move to drop loopback then we have no block device, so dm-verity is not usable (but fs-verity still is).

I think in all near term scenarios the erofs instance is accessing something that looks to it like a block device. The loopback discussion is just about whether that loopback is visible to userspace or not. EDIT: and to elaborate on this, I could imagine that we pass something to the kernel that is actually a dm-verity setup in a standardized format and then erofs calls into kernel signature verification layers?

[alexlarsson]

But I do also lean towards having some standardized flow for this. Hmm...actually, since EROFS reserves the first 1k (right?) for empty space for BIOS MBR-related compat reasons (which is kind of silly that we're doing that even with cfs, but that's an aside) - we could stick a signature there?

We already use this space to add a composefs header. However, we can't have a signature in there because it would be recursively affecting the fs-verity digest of the image file itself.

[hsiangkao]

which is kind of silly
you could leave anything here (even a pseudo tar header), since a lot of format starting from 0 (like squashfs), I don't think it's flexible (does anyone care 1k waste?) if you consider mering another format.

[alexlarsson]

Also, I had a very quick look at how dm-verity does signatures, and to me (admittedly an amateur here) it looks very similar (bordering on identical) to how the fs-verity built-in signatures work. They both just call the internal verify_pkcs7_signature() kernel function. If the fs-verity one is bad, why is the dm-verity one different?

@ebiggers could you in some short words describe the problems with the current fs-verity built-in signature support?

[cgwalters]

I'm not named Eric, but I have seen him comment on this. I think one problem is that today several properties are system-global (the sysctl and the keyring), which is inflexible. The global keyring is similar to dm-verity.

But the sysctl part is a bit problematic because if enabled and set to 1 basically would block all use of fs-verity for lesser-privileged system components because they can't change the keyring.

I could imagine mitigating this a bit by having a new ioctl like FS_IOC_CHECK_SIGNATURE_VERITY that just checks the signature in-kernel. (Then I guess the sysctl would be a new CONFIG_FS_VERITY_SIGNATURE_GLOBAL_SYSCTL).

[cgwalters]

However, there are lots of open questions:

Where do you find/store the key?

We could standardize an xattr for this? [system|user].composefs-signature?

[alexlarsson]

Yeah, the global sysctl is bogus, that I agree. We can't use that. However, being able to check that a file has a signature and then relying on the kernel to then validate it isn't completely shit.

I don't think we need FS_IOC_CHECK_SIGNATURE_VERITY. I think the proposed generic "validate data against pkcs#7 blob (and optional keyring)" syscall that @dhowells talked about is generally more useful.

[ebiggers]

Hi,

dm-verity is usually used with userspace signature verification, and until recently that was the only option. dm-verity's in-kernel signature verification was a recent addition for specialized use cases.

However, there are lots of open questions:

Where do you find/store the key?
How do you trust that someone didn't put in their own public key?
Can we do trust chains? key aging? revocation?
What file formats should it use?
What algorithms should it use?

You have all these same problems even with the in-kernel signature verification! How are you solving them?

I could imagine mitigating this a bit by having a new ioctl like FS_IOC_CHECK_SIGNATURE_VERITY that just checks the signature in-kernel.

I think that's a very bad idea. It would increase the kernel's attack surface for no real gain, since the API would just do some math that could easily be done in userspace instead. Userspace even has the benefit of being able to use any cryptographic library, whereas the kernel is much more limited in what it can do. The kernel has also made some poor design choices (e.g. reliance on X.509 and PKCS#7) which userspace is not limited to.

I am trying to understand the psychology behind why people think the built-in signatures are a good idea. I think what is going on is that people just do not know what to do, but they do "know" that their solution will involve both fs-verity (or dm-verity) and signatures. Then they see these built-in signature features and see they involve both of those components, so they think that must solve their problem for them.

I would like to help people understand that userspace signature verification is another solution that in most cases is better. Maybe people are looking for example code in fsverity-utils? Or an explanation in the docs that is much more "verbose" than the current ones, so that when people grep or Google for "fsverity signatures", something else comes up. I really don't know at this point. I see that ostree actually implements the signing for the built-in signatures itself, without using libfsverity, so clearly you do know how to use userspace cryptographic libraries; not knowing how to do so is not the problem here...

[cgwalters]

I am trying to understand the psychology behind why people think the built-in signatures are a good idea.

Well, part of the "philosophy" of composefs today (see the latest README at the toplevel) is that it's just combining existing things in a slightly novel way. The desire here is to reuse as much as possible logic from something we already depend on - and of those 3 things (overlayfs, erofs, fs-verity), it's just fsverity today that does signatures directly (with the valid caveats you mention). Also, all 3 of these things are primarily kernel constructions which we're just "gluing together" in userspace. (Though fsverity has a userspace part, which leads into the below...)

If you want to inspect the composefs metadata, there's already many tools for erofs - we don't need new ones. And those are implicitly shared with other use cases.

Another way to say this is that IMO it's an important goal for composefs is to be pretty widely applicable across many use cases, and the more external dependencies there are, it gets a bit harder.

But yes, you are absolutely right to point out that the default expectation for dm-verity has historically been verification in cryptsetup. Although...the more I dig into this, the more it looks like the push has been to move to using the in-kernel dm-verity signature logic - are you sure about your "specialized use cases" argument there? See e.g. https://gitlab.com/cryptsetup/cryptsetup/-/commit/f247038e6537dbe133d590097664bd81e9153b66 and it sure looks to me like the cryptsetup userspace effectively swapped to do verification in kernel by default if available. Similarly, the logic in systemd just calls this function if it exists, and only "falls back" to userspace verification: https://github.com/systemd/systemd/blob/34c4496ef2711d2a924e6f88fe3ff31cda080115/src/shared/dissect-image.c#L2549

(Although admittedly it's the same people doing both these things)

so they think that must solve their problem for them.

I wouldn't say must - we're debating architecture.

I would like to help people understand that userspace signature verification is another solution that in most cases is better.

What would you say to the people who did the veritysetup commit to move the logic to use the kernel?

Maybe people are looking for example code in fsverity-utils?

Perhaps a bit more than that; I could imagine "standardizing" e.g. an xattr for the signature at least? That only helps a little of course.

OK now that I look I see we're reiterating some bits of https://github.com/ebiggers/fsverity-utils#using-builtin-signatures here.

Anyways, I think everyone is in agreement that composefs signatures should work without CONFIG_FS_VERITY_BUILTIN_SIGNATURES enabled at build time (and without the sysctl enabled at runtime for sure).

[ebiggers]

I'm working on a kernel documentation patch that should clear things up a bit. After that I'd like to update fsverity-utils to add some example code for creating and verifying signatures.

Regardless, I would gently suggest that verifying signatures in userspace is a fairly small addition when you're already generating the signatures themselves, managing the key pairs, defining and enforcing a policy for which files have fs-verity enabled with a signature, re-implementing the fs-verity digest computation in userspace, etc. (FWIW, some of this could have used libfsverity. I'm not sure why that wasn't used, if there is really such a strong desire to reuse existing components.)

Android and Chrome OS both use dm-verity and don't use the in-kernel signature verification. The support for dm-verity in-kernel signature verification was pushed by Microsoft several years ago. I gave feedback on the original code review that the feature didn't seem well motivated. Their answer was basically that it is only intended for tightly locked down systems where even the process that sets up the dm-verity device (that has CAP_SYS_ADMIN) can't be trusted. It did get added anyway, though. I think it has suffered from a similar problem as the fs-verity builtin signatures, where some people who really should just be using userspace signature verification use the in-kernel support instead, because it seems like the "official" solution.

[cgwalters]

I gave feedback on the original code review that the feature didn't seem well motivated.

Thanks for that reference! The reply here is interesting I think:
https://lore.kernel.org/dm-devel/[email protected]/t/#u

They talk about keys "builtin to the kernel" but one thing I don't quite get is that most systems like this are going to be using UKIs (or something like it) i.e. kernel-with-compiled-in-initramfs, and one can just embed keys and userspace tooling in there and have them covered by the same Secure Boot or equivalent flow that would cover the builtin kernel keyring keys. But also on some of these "embedded-style" systems I've seen they try to avoid an initramfs at all (even one compiled in); maybe that's part of it. I guess at this point I/we should be asking them, not on this thread.

One thing I actually do agree with James Morris on here is that it is definitely desirable to support "strong binding" of the validation of a signature with other kernel objects like LSMs; this is related to what https://microsoft.github.io/ipe/ is doing as I understand it. So if in the composefs case, if an in-kernel code (e.g. fsverity using kernel keyring) attached metadata to an inode that said its signature was verified, and then we get that up through the layers from erofs to the overlayfs, one could use an LSM in a much more flexible way than the fsverity sysctl - e.g. enforce that some domains/processes can only open files or execute code from filesystem mounts that have a specific label. (Some handwaving here on my part, I haven't dug into IPE in depth)

Anyways, thanks for the reply. I am convinced for now at least that it makes sense for the ostree use case to support reusing its existing signature infrastructure (as crummy as it is, but that's another story...). (edit: though, I also would like to drain some of what's in ostree down into composefs, and so the question of how "composefs" handles signatures is still very relevant, and it should for sure not involve gpg)

[ebiggers]

fsverity: improve documentation for builtin signature support is the kernel documentation patch that I'm proposing.

[ebiggers]

One thing I actually do agree with James Morris on here is that it is definitely desirable to support "strong binding" of the validation of a signature with other kernel objects like LSMs; this is related to what https://microsoft.github.io/ipe/ is doing as I understand it.

IMA already supports this with fsverity since kernel v5.19. It's possible to e.g. use an IMA rule that only allows files with a particular SELinux type to be executed if there is a matching signature of the file's fsverity digest stored in the file's "security.ima" extended attribute. Note, this does not use the fsverity "builtin signatures" but rather the IMA signature tooling.

So if in the composefs case, if an in-kernel code (e.g. fsverity using kernel keyring) attached metadata to an inode that said its signature was verified, and then we get that up through the layers from erofs to the overlayfs, one could use an LSM in a much more flexible way than the fsverity sysctl - e.g. enforce that some domains/processes can only open files or execute code from filesystem mounts that have a specific label. (Some handwaving here on my part, I haven't dug into IPE in depth)

... though in the "composefs" use case, due to the stacking involved to get to the underlying EROFS image that is being authenticated, there would be no way for either IMA or IPE to make a decision about a file being executed on the overlayfs based on the authentication of the EROFS. The IPE patchset proposes a way to make decisions about files based on the authentication of the block device containing them, but that would not be enough here as it would not work with overlayfs. It would also need dm-verity, as fsverity + loopback would not work.

(I don't know whether IPE will be accepted upstream, as it duplicates a lot of functionality with IMA.)

[hsiangkao]

fsverity + loopback would not work

Sorry about that, it is too much background for me to fully look into the IPE thread.

I don't know if there is some barrier to run fsverity on loopback-devices, the only practical limitation is that fsverity doesn't support direct I/O yet, but loopback devices use buffered I/O by default, and I think fsverity direct I/O can be supported without diffculty in principle.

Am I missing something here? really I don't quite follow some status of IPE but I wonder if there could be some barrier or not, thanks.

[alexlarsson]

Ok, so for ostree usecase in particular, we already have code like ostree_sign_ed25519_data().
So, a minimal approach would be:

• During commit, pass the composefs fs-verity digest into ostree_sign_ed25519_data(), with a given key (similar to how we now compute the signature)
• At deploy time, extract this signature as .ostree.cfs.sig
• At boot time, load a public key from a know location in the initrd (have default location, over-ridable with kernel cmdline option).
• Get fs-verity digest from target composefs file, pass that and public key to ostree_sign_ed25519_data_verify()

One issue here is that ostree_sign_ed25519_data_verify() is in libostree and relies on glib, which are not normally linked into ostree-prepare-root. Maybe we can change that? Or just extract the right code.

Note: this code is mainly a front for libsodium