MastForest serialization (#1370)

* Introduce `ExternalNode` * Replace `Assembler.node_id_by_digest` map * add TODOP * Add `Host::get_mast_forest` * Move kernel and entrypoint out of `MastForest` * Remove ProgramError * docs * cleanup Program constructors * fix docs * Make `Program.kernel` an `Arc` * fix executable * invoke_mast_root: fix external node creation logic * add failing test * don't make root in `combine_mast_node_ids` and `compile_body` * fix External docs * fmt * fix `entrypoint` doc * Rename `Program::new_with_kernel()` * Document `MastForestStore` and `MemMastForestStore` * fix syscall * execute_* functions: use `MastForest` * `Program`: Remove `Arc` around kernel * remove `Arc` around `MastForest` in `Program` * Return error on malformed host * Simplify `DefaultHost` * `MastForest::add_node()`: add docs * fmt * add failing `duplicate_procedure()` test * Introduce `MastForestBuilder` * Rename `mod tests` -> `testing` * add `duplicate_node()` test * changelog * Program: use `assert!()` instead of `debug_assert!()` * `MastForest::make_root()`: add assert * fmt * Serialization for `MastNodeId` * serialization for MastNode variants except basic block * MastForest serialization scaffolding * define `MastNodeType` constructor from `MastNode` * test join serialization of MastNodeType * `MastNodeType` serialization of split * Revert "serialization for MastNode variants except basic block" This reverts commit efc24fd. * add TODOP * impl Deserializable for `MastForest` (scaffold) * mast_node_to_info() scaffold * try_info_to_mast_node scaffold * Rename `EncodedMastNodeType` * add info module * encode operations into `data` field * decode operations * implement `BasicBlockNode::num_operations_and_decorators()` * OperationOrDecoratorIterator * basic block node: move tests in new file * operation_or_decorator_iterator test * Implement `Operation::with_opcode_and_data()` * encode decorators * implement `decode_decorator()` * fix exec invocation * no else blk special case * add procedure roots comment * implement forgotten `todo!()` * `serialize_deserialize_all_nodes` test * `decode_operations_and_decorators`: fix bit check * confirm_assumptions test scaffold * minor adjustments * Introduce `StringTableBuilder` * naming * test confirm_operation_and_decorator_structure * remove TODOP * remove unused `MastNode::new_dyncall()` * Remove `Error` type * add TODOP * complete test `serialize_deserialize_all_nodes` * check digest on deserialization * remove TODOP * safely decode mast node ids * use method syntax in `MastNodeType` decoding * TODOPs * rewrite <= expression * new `MastNodeType` * implement `Deserializable` for `MastNodeType` * migrate tests to new * Use new MastNodeType * rename string_table_builder_ module * implement `BasicBlockDataBuilder` * add TODOP * BasicBlockDataDecoder * use `BasicBlockDataDecoder` * add headers * add `MastNodeInfo` method * return `Result` instead of `Option` * Remove TODOP * docs * chore: add section separators and fix typos * refactor: change type of the error code of u32assert2 from Felt to u32 (#1382) * impl `Serializable` for `Operation` * impl Deserializable for `Operation` * `StringTableBuilder`: switch to using blake 3 * `EncodedDecoratorVariant`: moved discriminant bit logic to `discriminant()` method * Remove basic block offset * Cargo: don't specify patch versions * make deserialization more efficient * num-traits and num-derive: set default-features false * Remove `OperationData` * `StringRef`: move string length to data buffer * store offset in block * Use `source.read_u32/u64()` * Update `MastNodeInfo` docstring * rename arguments in `encode_u32_pair` * Use basic block offset in deserialization * `BasicBlockDataDecoder`: use `ByteReader::read_u16/32()` methods * `StringTableBuilder`: fix comment * Remove `StringRef` in favor of `DataOffset` * cleanup `MastNodeType` serialization * derive `Copy` for `MastNodeType` * `MastNodeType` tests * add `MastNodeType` tests * use assert * fix asserts * changelog * fix docs --------- Co-authored-by: Bobbin Threadbare <[email protected]> Co-authored-by: Andrey Khmuro <[email protected]>
0xPolygonMiden · Jul 17, 2024 · b433b57 · b433b57
1 parent 56ccf86
commit b433b57
Show file tree

Hide file tree

Showing 20 changed files with 1,889 additions and 375 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -16,6 +16,7 @@
 - Relaxed the parser to allow one branch of an `if.(true|false)` to be empty
 - Added support for immediate values for `u32and`, `u32or`, `u32xor` and `u32not` bitwise instructions (#1362).
 - Optimized `std::sys::truncate_stuck` procedure (#1384).
+- Add serialization/deserialization for `MastForest` (#1370)
 
 #### Changed
 

diff --git a/air/src/constraints/stack/op_flags/mod.rs b/air/src/constraints/stack/op_flags/mod.rs
@@ -840,7 +840,7 @@ impl<E: FieldElement> OpFlags<E> {
     /// Operation Flag of U32ASSERT2 operation.
     #[inline(always)]
     pub fn u32assert2(&self) -> E {
-        self.degree6_op_flags[get_op_index(Operation::U32assert2(ZERO).op_code())]
+        self.degree6_op_flags[get_op_index(Operation::U32assert2(0).op_code())]
     }
 
     /// Operation Flag of U32ADD3 operation.

diff --git a/assembly/src/assembler/instruction/mod.rs b/assembly/src/assembler/instruction/mod.rs
@@ -119,17 +119,17 @@ impl Assembler {
             // ----- u32 manipulation -------------------------------------------------------------
             Instruction::U32Test => span_builder.push_ops([Dup0, U32split, Swap, Drop, Eqz]),
             Instruction::U32TestW => u32_ops::u32testw(span_builder),
-            Instruction::U32Assert => span_builder.push_ops([Pad, U32assert2(ZERO), Drop]),
+            Instruction::U32Assert => span_builder.push_ops([Pad, U32assert2(0), Drop]),
             Instruction::U32AssertWithError(err_code) => {
-                span_builder.push_ops([Pad, U32assert2(Felt::from(err_code.expect_value())), Drop])
+                span_builder.push_ops([Pad, U32assert2(err_code.expect_value()), Drop])
             }
-            Instruction::U32Assert2 => span_builder.push_op(U32assert2(ZERO)),
+            Instruction::U32Assert2 => span_builder.push_op(U32assert2(0)),
             Instruction::U32Assert2WithError(err_code) => {
-                span_builder.push_op(U32assert2(Felt::from(err_code.expect_value())))
+                span_builder.push_op(U32assert2(err_code.expect_value()))
             }
-            Instruction::U32AssertW => u32_ops::u32assertw(span_builder, ZERO),
+            Instruction::U32AssertW => u32_ops::u32assertw(span_builder, 0),
             Instruction::U32AssertWWithError(err_code) => {
-                u32_ops::u32assertw(span_builder, Felt::from(err_code.expect_value()))
+                u32_ops::u32assertw(span_builder, err_code.expect_value())
             }
 
             Instruction::U32Cast => span_builder.push_ops([U32split, Drop]),

diff --git a/assembly/src/assembler/instruction/u32_ops.rs b/assembly/src/assembler/instruction/u32_ops.rs
@@ -6,7 +6,6 @@ use crate::{
 use vm_core::{
     AdviceInjector, Felt,
     Operation::{self, *},
-    ZERO,
 };
 
 /// This enum is intended to determine the mode of operation passed to the parsing function
@@ -45,7 +44,7 @@ pub fn u32testw(span_builder: &mut BasicBlockBuilder) {
 ///
 /// Implemented by executing `U32ASSERT2` on each pair of elements in the word.
 /// Total of 6 VM cycles.
-pub fn u32assertw(span_builder: &mut BasicBlockBuilder, err_code: Felt) {
+pub fn u32assertw(span_builder: &mut BasicBlockBuilder, err_code: u32) {
     #[rustfmt::skip]
     let ops = [
         // Test the first and the second elements
@@ -171,7 +170,7 @@ pub fn u32not(span_builder: &mut BasicBlockBuilder) {
     let ops = [
         // Perform the operation
         Push(Felt::from(u32::MAX)),
-        U32assert2(ZERO),
+        U32assert2(0),
         Swap,
         U32sub,
 

diff --git a/core/Cargo.toml b/core/Cargo.toml
@@ -19,19 +19,14 @@ doctest = false
 
 [features]
 default = ["std"]
-std = [
-    "miden-crypto/std",
-    "miden-formatting/std",
-    "math/std",
-    "winter-utils/std",
-    "thiserror/std",
-]
+std = ["miden-crypto/std", "miden-formatting/std", "math/std", "winter-utils/std", "thiserror/std"]
 
 [dependencies]
 math = { package = "winter-math", version = "0.9", default-features = false }
-#miden-crypto = { version = "0.9", default-features = false }
 miden-crypto = { git = "https://github.com/0xPolygonMiden/crypto", branch = "next", default-features = false }
 miden-formatting = { version = "0.1", default-features = false }
+num-derive = { version = "0.4", default-features = false }
+num-traits = { version = "0.2", default-features = false }
 thiserror = { version = "1.0", git = "https://github.com/bitwalker/thiserror", branch = "no-std", default-features = false }
 winter-utils = { package = "winter-utils", version = "0.9", default-features = false }
 

diff --git a/core/src/mast/mod.rs b/core/src/mast/mod.rs
@@ -6,8 +6,11 @@ use miden_crypto::hash::rpo::RpoDigest;
 mod node;
 pub use node::{
     get_span_op_group_count, BasicBlockNode, CallNode, DynNode, ExternalNode, JoinNode, LoopNode,
-    MastNode, OpBatch, SplitNode, OP_BATCH_SIZE, OP_GROUP_SIZE,
+    MastNode, OpBatch, OperationOrDecorator, SplitNode, OP_BATCH_SIZE, OP_GROUP_SIZE,
 };
+use winter_utils::{ByteReader, ByteWriter, Deserializable, DeserializationError, Serializable};
+
+mod serialization;
 
 #[cfg(test)]
 mod tests;
@@ -18,6 +21,9 @@ pub trait MerkleTreeNode {
     fn to_display<'a>(&'a self, mast_forest: &'a MastForest) -> impl fmt::Display + 'a;
 }
 
+// MAST NODE ID
+// ================================================================================================
+
 /// An opaque handle to a [`MastNode`] in some [`MastForest`]. It is the responsibility of the user
 /// to use a given [`MastNodeId`] with the corresponding [`MastForest`].
 ///
@@ -27,14 +33,49 @@ pub trait MerkleTreeNode {
 #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
 pub struct MastNodeId(u32);
 
+impl MastNodeId {
+    /// Returns a new `MastNodeId` with the provided inner value, or an error if the provided
+    /// `value` is greater than the number of nodes in the forest.
+    ///
+    /// For use in deserialization.
+    pub fn from_u32_safe(
+        value: u32,
+        mast_forest: &MastForest,
+    ) -> Result<Self, DeserializationError> {
+        if (value as usize) < mast_forest.nodes.len() {
+            Ok(Self(value))
+        } else {
+            Err(DeserializationError::InvalidValue(format!(
+                "Invalid deserialized MAST node ID '{}', but only {} nodes in the forest",
+                value,
+                mast_forest.nodes.len(),
+            )))
+        }
+    }
+}
+
 impl fmt::Display for MastNodeId {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         write!(f, "MastNodeId({})", self.0)
     }
 }
 
+impl Serializable for MastNodeId {
+    fn write_into<W: ByteWriter>(&self, target: &mut W) {
+        self.0.write_into(target)
+    }
+}
+
+impl Deserializable for MastNodeId {
+    fn read_from<R: ByteReader>(source: &mut R) -> Result<Self, DeserializationError> {
+        let inner = source.read_u32()?;
+
+        Ok(Self(inner))
+    }
+}
+
 // MAST FOREST
-// ===============================================================================================
+// ================================================================================================
 
 /// Represents one or more procedures, represented as a collection of [`MastNode`]s.
 ///
@@ -94,7 +135,7 @@ impl MastForest {
     /// Returns the [`MastNode`] associated with the provided [`MastNodeId`] if valid, or else
     /// `None`.
     ///
-    /// This is the faillible version of indexing (e.g. `mast_forest[node_id]`).
+    /// This is the failable version of indexing (e.g. `mast_forest[node_id]`).
     #[inline(always)]
     pub fn get_node_by_id(&self, node_id: MastNodeId) -> Option<&MastNode> {
         let idx = node_id.0 as usize;