Consider adding Verifier::new_with_extra_roots implementation to other platforms

[complexspaces]

The functionality of new_with_extra_roots is primarily useful for Linux/WASM/BSD platforms that don't have a consistent source of trusted CA root/anchors available. However, many private/internal applications often use their own private CAs instead of publicly issued ones. This seems like a use case we could support without much burden, even if those users might be better off making their own webpki-based verifier instead.

Implementation details

It's worth noting that the Apple and Windows code for this already exists in a near-drop in form. Android would require more work to make a TrustManager that combined certificates.

macOS/iOS

We can call SecTrustSetAnchorCertificates to add additional roots to the evaluation, and then call SecTrustSetAnchorCertificatesOnly with false to trust both the custom roots and the default OS-provided ones.

Windows

We can create a custom CERT_CHAIN_ENGINE_CONFIG and set cAdditionalStore to a custom, in-memory certificate store. This engine and store are then included in our call to CertGetCertificateChain.

Android

I believe this can be done by using a PKIXParameters. I think the best idea is to create a custom TrustManager class that considers the system Keystore's trustmanager and then a custom Keystore containing the user-provided roots. I'm not yet sure about the exact implementation strategy though since the Android X509 and PKIX APIs are a handful. This and this blog may be helpful references.

[blind-oracle]

I second that. This also makes development easier since when using MacOS I've spent some time figuring out why the new_with_extra_roots method is missing :)

[djc]

Note that this is blocking (built-in) support for rustls-platform-verifier in reqwest: seanmonstar/reqwest#2286 (comment).

[complexspaces]

@stormshield-gt Based on rustls/rustls-native-certs#3 (comment), I believe this is the issue would be good to help with to forward reqwest integration.

If you'd like to take my existing work in https://github.com/rustls/rustls-platform-verifier/tree/shared/extra-roots-additions and update it to use the new types provided in rustls/webpki-roots#75, that would make progress on the iOS side (as macOS and iOS share a verifier impl).

[stormshield-gt]

Thanks for the hint, I will start here so.

[stormshield-gt]

I'm adding tests for the Verifier::new_extra_roots and I wonder if I should just replace the existing tests that override one CA. Does it make sense to keep them both ?

[complexspaces]

This is something I was also thinking about when working on the branch and I was undecided 😅. If its cleaner with tests to merge together the mock CA and extra roots, go ahead and do so.

The one important part we need to keep around though is the ability to tell the system verifier to only use those roots during testing, and not consider the wider system state. This isn't wanted in production but during testing its very valuable to keep things reproducible given the background-managed nature of trust roots on platforms.

[stormshield-gt]

I think we should reopen this while we don't support android and windows

[stormshield-gt]

I wonder if we should consider adding an option to configure if the extra roots must be exclusive or not. I mean if we should consider only the extra root or the extra root and the OS root store.

Reqwest has this flag in its client builder and if we want to integrate with it, I think we want to expose this functionality.

For instance, on MacOS, when reqwest use the native-tls feature, this flag is just propagated to secure transport crate which is then just used as an argument to the call of the SecTrustSetAnchorCertificatesOnly function.

We could easily replicate this in our implementation. Besides it will remove the need for having special field inside the Verifier only for testing.

[complexspaces]

It seems like an edge case, honestly, but maybe we do need to support it for reqwest to be happy.

IMO that flag doesn't make much sense to have in a platform verifier: why would you use this crate and only custom roots? You could just use rustls on its own with the webpki-based verifier if the roots are known to be good ahead-of-time. Usually the platform verifiers are better at handling "messy" certificate chains because of how much compat work the OSes put in since their code has to "just work."

[stormshield-gt]

I see your point in the case you want to use the extra root also on production, I also think you better off using the webpki verifier.

But in a case you use extra roots only for testing, I think it can be valuable to have the same verifier for testing and production, as the behavior might be slightly different.
As for this crate, when doing the test you might want only the extra root for reproducibility.

In any case, as you said, we might just need this for reqwest support.

[complexspaces]

The callout about end user testing is something I hadn't considered, thanks for mentioning that. I can see how people would want to maybe do similar to us and isolate unit test to an exclusive root store and not let the OS do any certificate fetching on-the-fly etc for reproducibility.

[cpu]

cpu closed this as completed in #135 now

GitHub misunderstood "partially fix 135" as "fix 135" - I believe all that remains now is support for this in the Android verifier.

[ofek]

Has this non-Android functionality been released yet?

[cpu]

No: v/0.3.4...main

[complexspaces]

I can put up a PR to start the release process. I think the new APIs have sat for long enough in main without anyone finding an obvious limitation right away.

[ofek]

That would be awesome, I'm very excited for reqwest to start using this!

[stormshield-gt]

Maybe before publishing we can take a look one more time at this #133 (comment) ?
As we now also iterate in the macOS/IOS case, maybe impl IntoIterator<Item = pki_types::CertificateDer<'static>> would be slightly more flexible ?

[djc]

Maybe before publishing we can take a look one more time at this #133 (comment) ? As we now also iterate in the macOS/IOS case, maybe impl IntoIterator<Item = pki_types::CertificateDer<'static>> would be slightly more flexible ?

Sounds good to me, want to submit a PR?

[stormshield-gt]

That's done with #145

[FrankenApps]

What is currently missing for this to be considered complete? Android support?

[complexspaces]

Correct, Android is the last sticking point. Its platform verifier APIs are pretty obtuse and poorly documented (especially on the behavior side, which is technically OSS but buried under several layered projects, class types, etc), so it needs some dedicated investigation. I have some high-level notes on this from fall last year (2024) that I didn't manage to go further on, but I'd be happy to bounce ideas around with another interested party. I'll paste the rest of the notes here cleaned up a bit:

The part I wasn't able to figure out at the time is how to nicely merge trust roots from the OS with those from the extra roots. At the system level there's two or three certificate kinds we need to account for:

• Default roots shipped with the system. To the best of my knowledge these are available in $ANDROID_ROOT/etc/security/cacerts on disk too. However I don't believe we can read this as the source of truth because Android lets you control trust per system CA in the settings and (AFAIK) doesn't account for personal vs work profile context.
• Roots manually added by the user into their system-wide trust settings. These do get written to disk, but the directory (/data/misc/keychain) isn't accessible to apps by default. They are available via a Keystore instance though.
• Per-user certificates provided via MDM for connections like internal servers or (in theory, but a stretch) SSL-based VPNs based ontop of rustls. I don't think these have been explicitly tested with the library yet, but one would hope they're also lumped into the AndroidCAStore Keystore instance that we use today.

The next design constraint is that we need to somehow pass this composite key store to a instance of PKIXParameters. This is primarily so we can keep a path open to fix #59 and (maybe) improve #66. It only has two constructors relevant here:

• A list of individual trust anchor objects
• A Keystore class instance.

This rules out (as far as I saw in my prior research) the ability to just implement a custom X509TrustManager class that checks two Keystore instances because we need to drop the existing X509TrustManager use in order to resolve #59 and some related warts.

Explained more shortly: We need both a way to access all three of these certificate kinds at once and tack on the contents of new_with_extra_roots. The Keystore-based constructor is highly preferable to avoid long-term maintenance issues as Android evolves and avoid performance problems from the AoT parsing of all system trust roots and independently checking their configured trust status (if such a thing is possible).

For anyone interested in exploring this, I would recommend exploring, to start, either:

• Seeing if its possible to make a joint Keystore.
• Experimenting with PKIXParameters::addCertStore to see if it has expected or desirable behavior as a "tacked on list" of custom certificates when the PKIXParameters were initialized from the system and/or testing mock Keystore.