Add support for decrypting S/MIME messages #11555

nitneuqr · 2024-09-06T17:01:31Z

As discussed on #6263, I'm opening this PR with an initial implementation of S/MIME decryption, in order to better discuss the API design, the algorithms we want to support, and how we want to approach testing.

My essential thoughts for testing were to do the round-trip: encryption using the PKCS7EnvelopeBuilder and decryption using the PKCS7EnvelopeDecryptor (or whatever its name will be). No tests were made against openssl, even though it might interesting.

For the Python API, I've tried to stick as much as possible with the current API design (PKCS7SignatureBuilder, PKCS7EnvelopeBuilder).

I'm new to rust, so please let me know if you see some issues in variable lifetime, or some unnecessary copying between Python & Rust.

During development, I've realized that we could migrate PKCS7 unpadding to rust instead of using types, so I did migrate it. I'll probably open another smaller PR for that matter (if that makes sense?). I still have some code using PKCS7 Unpadding on Python if needed.

cc @alex

alex · 2024-09-06T17:03:10Z

Thanks for working on this! I haven't had a chance to look in depth, but re: PKCS7Unpadder, it'd be great to have that in a stand-alone PR.

nitneuqr · 2024-09-06T17:15:50Z

Thanks for working on this! I haven't had a chance to look in depth, but re: PKCS7Unpadder, it'd be great to have that in a stand-alone PR.

Sure! Opened #11556 for that matter.

nitneuqr · 2024-09-08T09:05:13Z

Rebased to main after integration of #11556. Should I split it again in smaller PRs? Maybe around symmetric_decrypt or other subfunctions in the code.

alex · 2024-09-08T16:31:49Z

We're always happy to have smaller PRs split out (though it can sometimes be complicated due to coverage). I'm hoping to have time to review this today, though it might be tomorrow.

src/cryptography/hazmat/primitives/serialization/pkcs7.py

alex · 2024-09-10T03:50:04Z

@facutuesca FYI, if you're interested

nitneuqr · 2024-09-26T07:21:28Z

@alex @reaperhulk took into account the first comments and adapted the tests / coverage accordingly.

Do you have any time for another review? Do you need anything else from me before reviewing? 😊

alex · 2024-10-05T19:40:15Z

Sorry I got behind, will have a look now. Thanks for your patience!

alex

Did an initial review of the code -- I haven't really sat down and reviewed the core PKCS#7 logic against hte RFC yet though.

alex · 2024-10-05T19:42:19Z

src/cryptography/hazmat/primitives/serialization/pkcs7.py

+def pkcs7_decrypt_der(
+    data: bytes,
+    certificate: x509.Certificate,
+    private_key: rsa.RSAPrivateKey,
+    options: typing.Iterable[PKCS7Options],
+) -> bytes:
+    return _pkcs7_decrypt(
+        data, certificate, private_key, options, serialization.Encoding.DER
+    )
+
+
+def pkcs7_decrypt_pem(
+    data: bytes,
+    certificate: x509.Certificate,
+    private_key: rsa.RSAPrivateKey,
+    options: typing.Iterable[PKCS7Options],
+) -> bytes:
+    return _pkcs7_decrypt(
+        data, certificate, private_key, options, serialization.Encoding.PEM
+    )


I don't love needing to pass an encoding down, ideally we'd be able to PEM decode the data and then simply call pkcs7_decrypt_der with the results. That's probably annoying to do in Python though.

A few solutions:

Separating the PEM decode logic in another rust function, and calling it separately

Finding a library that does PEM to DER in Python

Coding it in-house (but probably annoying and / or dirty)

I'm for the 1st one, I think.

I've implemented a quick function in python using rust to decode pem to der. Please let me know what you think!

src/cryptography/hazmat/primitives/serialization/pkcs7.py

src/rust/src/pkcs12.rs

src/rust/src/pkcs7.rs

alex · 2024-10-05T19:47:19Z

src/rust/src/pkcs7.rs

+    };
+
+    // Deserialize the content info
+    let content_info = asn1::parse_single::<pkcs7::ContentInfo<'_>>(&extracted_data).unwrap();


@reaperhulk This would be our first time using our DER parser for PKCS7. On the one hand: new API, so there's no backwards-compat concerns. On the other hand, surely someone is emitting awful BER PKCS#7. Any concerns?

@reaperhulk just going to keep pinging you on this :-) (Enjoy your vacation)

Let's assume we release with DER only parsing. In that scenario we're attempting to empirically determine whether or not there's enough BER out there to matter. If no one comes to complain then great, job done. However, if we discover we're wrong and BER is relevant, what is our plan? Are we going to add BER support in rust?

If we do determine that this API is basically low utility for many users because their S/MIME messages aren't valid DER, it seems we have a few options:

Declare that's how it goes. This is a net new API, so no users are worse off than they were before. May result in greater timelines for adoption of this, and migration away from other libraries.

Re-implement in terms of OpenSSL S/MIME APIs. Probably straightforward, although as-ever, assessing the compatibility surface will be a PITA.

Add BER support to rust-asn1. No fucking chance, not worth discussing.

We force ourselves to do a survey of the ecosystem before landing this so we can contact producers of non-DER S/MIMEs.

I have a soft spot for 1, I could be persuaded of 4, and if we really need to do it, I guess 2 is the way to go.

WDYT?

Does Go support this? They're DER only for whatever set of PKCS7 they support right?

Go stdlib doesn't support PKCS#7 I don't think. I did find:

https://github.com/mozilla-services/pkcs7/blob/v0.9.0/pkcs7.go#L155-L160

https://github.com/InfiniteLoopSpace/go_S-MIME/blob/master/cms/protocol/contentinfo.go#L25

Sooooo

src/rust/src/pkcs7.rs

src/rust/src/test_support.rs

alex

@reaperhulk would still like your views on deserialization for PKCS#7 in Rust.

src/rust/src/pkcs7.rs

alex · 2024-10-21T03:40:05Z

src/rust/src/pkcs7.rs

+    // Return content in different form, based on options
+    let text_mode = options.contains(types::PKCS7_TEXT.get(py)?)?;
+    let plain_data = if options.contains(types::PKCS7_BINARY.get(py)?)? {
+        plain_content
+    } else {
+        let decanonicalized = smime_decanonicalize(plain_content.as_bytes(), text_mode);
+        pyo3::types::PyBytes::new_bound(py, decanonicalized.into_owned().as_slice())
+    };


Note to self: Think about this more.

src/rust/src/test_support.rs

vectors/cryptography_vectors/pkcs7/rsa_ca.pem

alex · 2024-10-28T16:13:46Z

(Thanks for updating, I'm going to pause on reviewing until #11843 is merged, hopefully that'll be quick.)

first round-trip tests feat: made asn1 structures readable refacto: adapted existing functions accordingly feat/pkcs12: added symmetric_decrypt feat: deserialize 3 possible encodings feat: handling AES-128 feat: raise error when no recipient is found feat/pkcs7: added decanonicalize function feat/asn1: added decode_der_data feat/pkcs7: added smime_enveloped_decode tests are the round-trip (encrypt & decrypt) more tests for 100% python coverage test support pkcs7_encrypt with openssl added algorithm to pkcs7_encrypt signature refacto: decrypt function is clearer flow is more natural refacto: added all rust error tests refacto: added another CA chain for checking fix: const handling Refactor PKCS7Decryptor to pkcs7_decrypt refacto: removed SMIME_ENVELOPED_DECODE from rust code refacto: removed decode_der_data adapted tests accordingly removed the PEM tag check added tests for smime_decnonicalize one more test case Update src/rust/src/pkcs7.rs Co-authored-by: Alex Gaynor <[email protected]> took comments into account pem to der is now outside of decrypt fix: removed test_support pkcs7_encrypt added vector for aes_256_cbc encrypted pkcs7 feat: not using test_support decrypt anymore added new vectors for PKCS7 tests feat: using pkcs7 vectors removed previous ones fix: changed wrong function feat: added certificate issuer check test: generating the RSA chain removed the vectors accordingly

nitneuqr · 2024-11-06T08:37:05Z

#11843 is merged, I've rebased the branch onto main to take the new changes into account. It should b ready to review again!

alex · 2024-11-06T12:08:00Z

Great, will try to get to it this evening. Thanks!

…

On Wed, Nov 6, 2024 at 3:37 AM Quentin Retourne ***@***.***> wrote: #11843 is merged, I've rebased the branch onto main to take the new changes into account. It should b ready to review again! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

alex · 2024-11-08T05:05:58Z

src/rust/cryptography-x509/src/pkcs7.rs

 pub struct SignedData<'a> {
    pub version: u8,
-    pub digest_algorithms: asn1::SetOfWriter<'a, common::AlgorithmIdentifier<'a>>,
+    pub digest_algorithms: common::Asn1ReadableOrWritable<


Would you be up for pulling the Asn1ReadableOrWritable pieces out into a separate PR? I think this is very very close, and I'm looking for ways to keep it moving as I review the last pieces.

Done, #11922 is opened!

nitneuqr mentioned this pull request Sep 6, 2024

Rustify PKCS7 unpadding #11556

Merged

nitneuqr force-pushed the smime-decryption branch 2 times, most recently from cb082ce to e1c4620 Compare September 8, 2024 09:01

alex reviewed Sep 10, 2024

View reviewed changes

src/cryptography/hazmat/primitives/serialization/pkcs7.py Outdated Show resolved Hide resolved

nitneuqr force-pushed the smime-decryption branch from d4e7741 to 69d2e23 Compare September 12, 2024 20:44

nitneuqr requested review from alex and reaperhulk September 18, 2024 08:46

alex reviewed Oct 5, 2024

View reviewed changes

alex reviewed Oct 21, 2024

View reviewed changes

nitneuqr force-pushed the smime-decryption branch from 1591c66 to 0dea78d Compare October 27, 2024 12:26

nitneuqr mentioned this pull request Oct 27, 2024

added new vectors for PKCS7 tests #11843

Merged

nitneuqr force-pushed the smime-decryption branch from e504553 to fd8db30 Compare November 6, 2024 08:34

alex reviewed Nov 8, 2024

View reviewed changes

nitneuqr mentioned this pull request Nov 8, 2024

passing PKCS7 Content Info to readable #11922

Merged

nitneuqr added 2 commits November 8, 2024 16:11

moved symmetric_decrypt to pkcs7.rs

9eed863

went too fast ^^

91b014c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for decrypting S/MIME messages #11555

Add support for decrypting S/MIME messages #11555

nitneuqr commented Sep 6, 2024

alex commented Sep 6, 2024

nitneuqr commented Sep 6, 2024

nitneuqr commented Sep 8, 2024

alex commented Sep 8, 2024

alex commented Sep 10, 2024

nitneuqr commented Sep 26, 2024

alex commented Oct 5, 2024

alex left a comment

alex Oct 5, 2024

nitneuqr Oct 12, 2024

nitneuqr Oct 12, 2024

alex Oct 5, 2024

alex Nov 8, 2024

reaperhulk Nov 8, 2024

alex Nov 8, 2024

reaperhulk Nov 8, 2024

alex Nov 8, 2024

alex left a comment

alex Oct 21, 2024

alex commented Oct 28, 2024

nitneuqr commented Nov 6, 2024

alex commented Nov 6, 2024 via email

alex Nov 8, 2024

nitneuqr Nov 8, 2024 •

edited

Loading

Add support for decrypting S/MIME messages #11555

Are you sure you want to change the base?

Add support for decrypting S/MIME messages #11555

Conversation

nitneuqr commented Sep 6, 2024

alex commented Sep 6, 2024

nitneuqr commented Sep 6, 2024

nitneuqr commented Sep 8, 2024

alex commented Sep 8, 2024

alex commented Sep 10, 2024

nitneuqr commented Sep 26, 2024

alex commented Oct 5, 2024

alex left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alex left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alex commented Oct 28, 2024

nitneuqr commented Nov 6, 2024

alex commented Nov 6, 2024 via email

Choose a reason for hiding this comment

nitneuqr Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

nitneuqr Nov 8, 2024 •

edited

Loading