Merge pull request #35 from xmtp/nm/merge-latest-updates

Merge upstream (Aug 19th, 2024)
xmtp · Aug 20, 2024 · 6048722 · 6048722
2 parents 9cb3207 + cf42738
commit 6048722
Show file tree

Hide file tree

Showing 167 changed files with 7,511 additions and 4,981 deletions.
diff --git a/.github/workflows/benches.yml b/.github/workflows/benches.yml
@@ -1,8 +1,8 @@
 name: Benchmarks
 
-concurrency: 
+concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true 
+  cancel-in-progress: true
 
 on:
   push:
@@ -32,3 +32,8 @@ jobs:
       - uses: Swatinem/rust-cache@v2
       - name: Benchmarks
         run: cargo bench -p openmls --verbose
+
+      - name: Large groups
+        run: |
+          cargo run -p openmls --example large-groups --release -- --write -g 2 3
+          cargo run -p openmls --example large-groups --release -- -g 3
diff --git a/.github/workflows/clippy.yml b/.github/workflows/clippy.yml
@@ -16,4 +16,7 @@ jobs:
         with:
           components: clippy
       - uses: Swatinem/rust-cache@v2
-      - run: cargo clippy -p openmls --tests -- -D warnings
+      - run: |
+          sudo apt-get -y install protoc-gen-go # Needed to build the interop client
+          echo $(go env GOPATH)/bin >> $GITHUB_PATH
+          cargo clippy -p openmls --tests --benches --examples -p openmls_basic_credential -p cli -p interop_client -p mls-ds -p ds-lib -p openmls_libcrux_crypto -p openmls_memory_storage -p openmls_rust_crypto -p openmls_test -p openmls-wasm -p openmls_traits -- -D warnings
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,10 +5,31 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## 0.6.0-pre.2 (2024-08-XX)
 
 ### Added
 
+- [#1639](https://github.com/openmls/openmls/pull/1639): Introduce `PublicStorageProvider` trait to independently allow for the storage of `PublicGroup` instances.
+- [#1641](https://github.com/openmls/openmls/pull/1641): Extend the `PublicGroup` API with `add_proposal()`, `remove_proposal()`, and `queued_proposals()`.
+
+### Changed
+
+- [#1637](https://github.com/openmls/openmls/pull/1637): Remove `serde` from `MlsGroup`.
+- [#1638](https://github.com/openmls/openmls/pull/1638): Remove `serde` from `PublicGroup`. `PublicGroup::load()` becomes public to load a group from the storage provider.
+- [#1642](https://github.com/openmls/openmls/pull/1642): `OpenMlsProvider` is no longer required for the `PublicGroup` API. The `PublicGroup` API now uses the `PublicStorageProvider` trait directly. `ProcessMessageError::InvalidSignature` was removed and replaced with `ValidationError::InvalidSignature`.
+
+### Removed
+
+
+### Fixed
+
+- [#1641](https://github.com/openmls/openmls/pull/1641): Fixed missing storage of queued proposals & clearing of the queued proposals.
+
+## 0.6.0-pre.1 (2024-07-22)
+
+### Added
+
+- [#1629](https://github.com/openmls/openmls/pull/1629): Add `add_members_without_update` function to `MlsGroup` to allow the creation of add-only commits
 - [#1506](https://github.com/openmls/openmls/pull/1506): Add `StagedWelcome` and `StagedCoreWelcome` to make joining a group staged in order to inspect the `Welcome` message. This was followed up with PR [#1533](https://github.com/openmls/openmls/pull/1533) to adjust the API.
 - [#1516](https://github.com/openmls/openmls/pull/1516): Add `MlsGroup::clear_pending_proposals` to the public API; this allows users to clear a group's internal `ProposalStore`
 - [#1565](https://github.com/openmls/openmls/pull/1565): Add new `StorageProvider` trait to the `openmls_traits` crate.
@@ -29,6 +50,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - [#1542](https://github.com/openmls/openmls/pull/1542): Add support for custom proposals. ProposalType::Unknown is now called ProposalType::Other. Proposal::Unknown is now called Proposal::Other.
 - [#1559](https://github.com/openmls/openmls/pull/1559): Remove the `PartialEq` type constraint on the error type of both the `OpenMlsRand` and `OpenMlsKeyStore` traits. Additionally, remove the `Clone` type constraint on the error type of the `OpenMlsRand` trait.
 - [#1565](https://github.com/openmls/openmls/pull/1565): Removed `OpenMlsKeyStore` and replace it with a new `StorageProvider` trait in the `openmls_traits` crate.
+- [#1606](https://github.com/openmls/openmls/pull/1606): Added additional `LeafNodeParameters` argument to `MlsGroup.self_update()` and `MlsGroup.propose_self_update()` to allow for updating the leaf node with custom parameters. `MlsGroup::join_by_external_commit()` now also takes optional parameters to set the capabilities and the extensions of the LeafNode.
+- [#1615](https://github.com/openmls/openmls/pull/1615): Changes the AAD handling. The AAD is no longer persisted and needs to be set before every API call that generates an `MlsMessageOut`. The functions `ProccessedMessage` to accees the AAD has been renamed to `aad()`.
 
 ### Fixed
 

diff --git a/Cargo.toml b/Cargo.toml
@@ -18,8 +18,8 @@ resolver = "2"
 
 # Central dependency management for some crates
 [workspace.dependencies]
-tls_codec = { version = "0.4.2-pre.1", features = [
+tls_codec = { version = "0.4.1", features = [
   "derive",
   "serde",
   "mls",
-], git = "https://github.com/rustcrypto/formats" }
+]}
diff --git a/basic_credential/Cargo.toml b/basic_credential/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "openmls_basic_credential"
-version = "0.2.0"
+version = "0.3.0-pre.1"
 authors = ["OpenMLS Authors"]
 edition = "2021"
 description = "A Basic Credential implementation for OpenMLS"
@@ -10,7 +10,7 @@ repository = "https://github.com/openmls/openmls/tree/main/basic_credential"
 readme = "README.md"
 
 [dependencies]
-openmls_traits = { version = "0.2.0", path = "../traits" }
+openmls_traits = { version = "0.3.0-pre.2", path = "../traits" }
 tls_codec = { workspace = true }
 serde = "1.0"
 

diff --git a/basic_credential/src/lib.rs b/basic_credential/src/lib.rs
@@ -137,6 +137,18 @@ impl SignatureKeyPair {
             .flatten()
     }
 
+    /// Delete a signature key pair from the key store.
+    pub fn delete<T: StorageProvider<CURRENT_VERSION>>(
+        store: &T,
+        public_key: &[u8],
+        signature_scheme: SignatureScheme,
+    ) -> Result<(), T::Error> {
+        let id = StorageId {
+            value: id(public_key, signature_scheme),
+        };
+        store.delete_signature_key_pair(&id)
+    }
+
     /// Get the public key as byte slice.
     pub fn public(&self) -> &[u8] {
         self.public.as_ref()

diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md
@@ -8,9 +8,11 @@
   - [Group configuration](user_manual/group_config.md)
   - [Creating groups](user_manual/create_group.md)
   - [Join a group from a Welcome message](user_manual/join_from_welcome.md)
+  - [Join a group from an External Commit message](user_manual/join_from_external_commit.md)
   - [Adding members to a group](user_manual/add_members.md)
   - [Removing members from a group](user_manual/remove_members.md)
   - [Updating own key package](user_manual/updates.md)
+  - [Using Additional Authenticated Data (AAD)](user_manual/aad.md)
   - [Leaving a group](user_manual/leaving.md)
   - [Custom proposals](user_manual/custom_proposals.md)
   - [Creating application messages](user_manual/application_messages.md)

diff --git a/book/src/app_validation.md b/book/src/app_validation.md
@@ -4,6 +4,64 @@
 >
 > **⚠️** This chapter is work in progress (see [#1504](https://github.com/openmls/openmls/issues/1504)).
 
+## Credential Validation
+
+### Acceptable Presented Identifiers
+
+> The application using MLS is responsible for specifying which identifiers
+> it finds acceptable for each member in a group. In other words, following
+> the model that [[RFC6125]] describes for TLS, the application maintains a list
+> of "reference identifiers" for the members of a group, and the credentials
+> provide "presented identifiers". A member of a group is authenticated by first
+> validating that the member's credential legitimately represents some presented
+> identifiers, and then ensuring that the reference identifiers for the member
+> are authenticated by those presented identifiers
+>
+> -- [RFC9420, Section 5.3.1](https://www.rfc-editor.org/rfc/rfc9420.html#section-5.3.1-1)
+>
+### Validity of Updated Presented Identifiers
+
+> In cases where a member's credential is being replaced, such as the Update and
+> Commit cases above, the AS MUST also verify that the set of presented identifiers
+> in the new credential is valid as a successor to the set of presented identifiers
+> in the old credential, according to the application's policy.
+>
+> -- [RFC9420, Section 5.3.1](https://www.rfc-editor.org/rfc/rfc9420.html#section-5.3.1-5)
+
+### Application ID is Not Authenticed by AS
+
+> However, applications MUST NOT rely on the data in an application_id extension
+> as if it were authenticated by the Authentication Service, and SHOULD gracefully
+> handle cases where the identifier presented is not unique.
+>
+> -- [RFC9420, Section 5.3.3](https://www.rfc-editor.org/rfc/rfc9420.html#section-5.3.3-6)
+
+## LeafNode Validation
+
+### Specifying the Maximum Total Acceptable Lifetime
+
+> Applications MUST define a maximum total lifetime that is acceptable for a
+> LeafNode, and reject any LeafNode where the total lifetime is longer than this
+> duration. In order to avoid disagreements about whether a LeafNode has a valid
+> lifetime, the clients in a group SHOULD maintain time synchronization (e.g.,
+> using the Network Time Protocol [[RFC5905]]).
+>
+> -- [RFC9420, Section 7.2](https://www.rfc-editor.org/rfc/rfc9420.html#section-7.2-10)
+
+## PrivateMessage Validation
+
+### Structure of AAD is Application-Defined
+
+> It is up to the application to decide what authenticated_data to provide and
+> how much padding to add to a given message (if any). The overall size of the
+> AAD and ciphertext MUST fit within the limits established for the group's AEAD
+> algorithm in [[CFRG-AEAD-LIMITS]].
+>
+> -- [RFC9420, Section 6.3.1](https://www.rfc-editor.org/rfc/rfc9420.html#section-6.3.1-11)
+
+Therefore, the application must also validate whether the AAD adheres to the
+prescribed format.
+
 ## Proposal Validation
 
 When processing a commit, the application has to ensure that the application
@@ -24,3 +82,7 @@ The RFC requires the following check
 
 Since OpenMLS does not know the relevant policies, the application MUST ensure
 that the credentials are checked according to the policy.
+
+[RFC6125]: https://www.rfc-editor.org/rfc/rfc6125.html
+[RFC5905]: https://www.rfc-editor.org/rfc/rfc5905.html
+[CFRG-AEAD-LIMITS]: https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-aead-limits-08
diff --git a/book/src/performance.md b/book/src/performance.md
@@ -39,9 +39,9 @@ It is the same scenario as the somewhat stable group but with a very small Y and
 
 In addition to the three scenarios above extreme and corner cases are interesting.
 
-### Every second leave is blank
+### Every second leaf is blank
 
-Only every second leave in the tree is non-blank.
+Only every second leaf in the tree is non-blank.
 
 ## Use Case Scenarios
 

diff --git a/book/src/traits/traits.md b/book/src/traits/traits.md
@@ -35,7 +35,7 @@ It simply needs to implement two functions to generate cryptographically secure
 randomness and store it in an array or vector.
 
 ```rust,no_run,noplayground
-{{#include ../../../traits/src/random.rs:8:16}}
+{{#include ../../../traits/src/random.rs:openmls_rand}}
 ```
 
 ### OpenMlsCrypto
@@ -48,27 +48,29 @@ This trait defines all cryptographic functions required by OpenMLS. In particula
 - Signatures
 - HPKE
 
-```rust,no_run,noplayground
-{{#include ../../../traits/src/crypto.rs:10}}
-```
-
 ### StorageProvider
 
 This trait defines an API for a storage backend that is used for all OpenMLS
 persistence.
 
-The store provides functions to `store`, `read`, and `delete` values.
-Note that it does not allow updating values.
-Instead, entries must be deleted and newly stored.
+The store provides functions for reading and updating stored values.
+Each sort of value has separate methods for accessing or mutating the state.
+In order to decouple the provider from the OpenMLS implementation, while still
+having legible types at the provider, there are traits that mirror all the types
+stored by OpenMLS. The provider methods use values constrained by these traits as
+as arguments.
 
 ```rust,no_run,noplayground
-{{#include ../../../traits/src/storage.rs:16:25}}
+{{#include ../../../traits/src/storage.rs:traits}}
 ```
 
-The trait is generic over a `VERSION`, which is used to ensure that the values
+The traits are generic over a `VERSION`, which is used to ensure that the values
 that are persisted can be upgraded when OpenMLS changes the stored structs.
 
-Every function takes `Key` and `Value` arguments.
+The traits used as arguments to the storage methods are constrained to implement
+the `Key` or `Entity` traits as well, depending on whether they are only used for
+addressing (in which case they are a `Key`) or whether they represent a stored
+value (in which case they are an `Entity`).
 
 ```rust,no_run,noplayground
 {{#include ../../../traits/src/storage.rs:key_trait}}
@@ -78,13 +80,6 @@ Every function takes `Key` and `Value` arguments.
 {{#include ../../../traits/src/storage.rs:entity_trait}}
 ```
 
-To ensure that each function takes the correct input, they use trait bounds.
-These are the available traits.
-
-```rust,no_run,noplayground
-{{#include ../../../traits/src/storage.rs:traits}}
-```
-
 An implementation of the storage trait should ensure that it can address and
 efficiently handle values.
 
@@ -127,20 +122,22 @@ fn write_key_package<
 This allows the application to iterate over the hash references and delete outdated
 key packages.
 
-### OpenMlsCryptoProvider
+### OpenMlsProvider
 
 Additionally, there's a wrapper trait defined that is expected to be passed into
 the public OpenMLS API.
 Some OpenMLS APIs require only one of the sub-traits, though.
 
 ```rust,no_run,noplayground
-{{#include ../../../traits/src/traits.rs:15:28}}
+{{#include ../../../traits/src/traits.rs:openmls_provider}}
 ```
 
 ## Implementation Notes
 
 It is not necessary to implement all sub-traits if one functionality is missing.
-Suppose you want to use a persisting key store. In that case, it is sufficient to do a new implementation of the key store trait and combine it with one of the provided crypto and randomness trait implementations.
+Suppose you want to use a persisting storage provider. In that case, it is
+sufficient to do a new implementation of the `StorageProvider` trait and
+combine it with one of the provided crypto and randomness trait implementations.
 
 [rust crypto]: https://crates.io/crates/openmls_rust_crypto
 [libcrux crypto]: https://crates.io/crates/openmls_libcrux_crypto

diff --git a/book/src/user_manual/README.md b/book/src/user_manual/README.md
@@ -5,7 +5,10 @@ The user manual describes how to use the different parts of the OpenMLS API.
 ## Prerequisites
 
 Most operations in OpenMLS require a `provider` object that provides all required cryptographic algorithms via the [`OpenMlsCryptoProvider`] trait.
-Currently, there are two implementations available through the [openmls_rust_crypto] crate.
+Currently, there are two implementations available:
+
+- one through the [openmls_rust_crypto] crate.
+- one through the [openmls_libcrux_crypto] crate.
 
 Thus, you can create the `provider` object for the following examples using ...
 
@@ -15,3 +18,4 @@ Thus, you can create the `provider` object for the following examples using ...
 
 [`openmlscryptoprovider`]: https://docs.rs/openmls/latest/openmls/prelude/trait.OpenMlsCryptoProvider.html
 [openmls_rust_crypto]: https://crates.io/crates/openmls_rust_crypto
+[openmls_libcrux_crypto]: https://crates.io/crates/openmls_libcrux_crypto
diff --git a/book/src/user_manual/aad.md b/book/src/user_manual/aad.md
@@ -0,0 +1,20 @@
+# Using Additional Authenticated Data (AAD)
+
+The Additional Authenticated Data (AAD) is a byte sequence that can be included in both private and public messages. By design, it is always authenticated (signed) but never encrypted. Its purpose is to contain data that can be inspected but not changed while a message is in transit.
+
+## Setting the AAD
+
+Members can set the AAD by calling the  `.set_aad()` function. The AAD will remain set until the next API call that successfully generates an `MlsMessageOut`. Until then, the AAD can be inspected with the `.aad()` function.
+
+```rust,no_run,noplayground
+{{#include ../../../openmls/tests/book_code.rs:set_aad}}
+```
+
+## Inspecting the AAD
+
+Members can inspect the AAD of an incoming message once the message has been processed. The AAD can be accessed with the `.aad()` function of a `ProcessedMessage`.
+
+```rust,no_run,noplayground
+{{#include ../../../openmls/tests/book_code.rs:inspect_aad}}
+```
+
diff --git a/book/src/user_manual/add_members.md b/book/src/user_manual/add_members.md
@@ -10,6 +10,12 @@ Members can be added to the group atomically with the `.add_members()` function.
 
 The function returns the tuple `(MlsMessageOut, Welcome)`. The `MlsMessageOut` contains a Commit message that needs to be fanned out to existing group members. The `Welcome` message must be sent to the newly added members.
 
+### Adding members without update
+
+The `.add_members_without_update()` function functions the same as the `.add_members()` function, except that it will only include an update to the sender's key material if the sender's proposal store includes a proposal that requires a path. For a list of proposals and an indication whether they require a `path` (i.e. a key material update) see [Section 17.4 of RFC 9420](https://www.rfc-editor.org/rfc/rfc9420.html#section-17.4).
+
+Not sending an update means that the sender will not achieve post-compromise security with this particular commit. However, not sending an update saves on performance both in terms of computation and bandwidth. Using `.add_members_without_update()` can thus be a useful option if the ciphersuite of the group features large public keys and/or expensive encryption operations.
+
 ## Proposal
 
 Members can also be added as a proposal (without the corresponding Commit message) by using the `.propose_add_member()` function:

diff --git a/book/src/user_manual/persistence.md b/book/src/user_manual/persistence.md
@@ -1,13 +1,11 @@
 # Persistence of Group Data
 
-The state of a given `MlsGroup` instance can be written or read at any time using the `.save()` or `.load()` functions respectively. The functions take as input a struct implementing either the `Write` (`.save()`) or `Read` (`.load()`) trait.
-
-Since some group operations might or might not change the `MlsGroup` state depending on the context, the group maintains the `state_changed` flag, which is set to `true` whenever the state is changed by an `MlsGroup` function. The state of the flag can be queried using the `.state_changed()` function.
-
-## Group Lockout Upon State Loss
-
-MLS provides strong Post-Compromise Security properties, which means that key material is regularly refreshed and old key material becomes stale very quickly. Consequently, regularly persisting state is important, especially after the client has created a commit or issued an Update proposal, thus introducing new key material into the group. A loss of state in such a situation is only recoverable in specific cases where the commit was rejected by the Delivery Service or if the proposed Update was not committed. A re-join is required in most cases to continue participating in a group after a loss of group state. To avoid a loss of state and the associated re-join, persisting `MlsGroup` state after each state-changing group operation is mandatory.
+The state of a given `MlsGroup` instance is continuously written to the configured
+`StorageProvider`. Later, the `MlsGroup` can be loaded from the provider using
+the `load` constructor, which can be called with the respective storage provider
+as well as the `GroupId` of the group to be loaded. For this to work, the group
+must have been written to the provider previously.
 
 ## Forward-Secrecy Considerations
 
-The `MlsGroup` state that is persisted using the `.save()` function contains private key material. As a consequence, the application needs to delete old group states to achieve Forward-Secrecy w.r.t. that key material. Since, as detailed above, an old group state is stale immediately after most group operations, we recommend deleting old group states as soon as a new one has been written.
+OpenMLS uses the `StorageProvider` to store sensitive key material. To achieve forward-secrecy (i.e. to prevent an adversary from decrypting messages sent in the past if a client is compromised), OpenMLS frequently deletes previously used key material through calls to the `StorageProvider`. `StorageProvider` implementations must thus take care to ensure that values deleted through any of the `delete_` functions of the trait are irrevocably deleted and that no copies are kept.