Skip to content

Commit

Permalink
MastForest serialization (#1370)
Browse files Browse the repository at this point in the history
* Introduce `ExternalNode`

* Replace `Assembler.node_id_by_digest` map

* add TODOP

* Add `Host::get_mast_forest`

* Move kernel and entrypoint out of `MastForest`

* Remove ProgramError

* docs

* cleanup Program constructors

* fix docs

* Make `Program.kernel` an `Arc`

* fix executable

* invoke_mast_root: fix external node creation logic

* add failing test

* don't make root in `combine_mast_node_ids` and `compile_body`

* fix External docs

* fmt

* fix `entrypoint` doc

* Rename `Program::new_with_kernel()`

* Document `MastForestStore` and `MemMastForestStore`

* fix syscall

* execute_* functions: use `MastForest`

* `Program`: Remove `Arc` around kernel

* remove `Arc` around `MastForest` in `Program`

* Return error on malformed host

* Simplify `DefaultHost`

* `MastForest::add_node()`: add docs

* fmt

* add failing `duplicate_procedure()` test

* Introduce `MastForestBuilder`

* Rename `mod tests` -> `testing`

* add `duplicate_node()` test

* changelog

* Program: use `assert!()` instead of `debug_assert!()`

* `MastForest::make_root()`: add assert

* fmt

* Serialization for `MastNodeId`

* serialization for MastNode variants except basic block

* MastForest serialization scaffolding

* define `MastNodeType` constructor from `MastNode`

* test join serialization of MastNodeType

* `MastNodeType` serialization of split

* Revert "serialization for MastNode variants except basic block"

This reverts commit efc24fd.

* add TODOP

* impl Deserializable for `MastForest` (scaffold)

* mast_node_to_info() scaffold

* try_info_to_mast_node scaffold

* Rename `EncodedMastNodeType`

* add info module

* encode operations into `data` field

* decode operations

* implement `BasicBlockNode::num_operations_and_decorators()`

* OperationOrDecoratorIterator

* basic block node: move tests in new file

* operation_or_decorator_iterator test

* Implement `Operation::with_opcode_and_data()`

* encode decorators

* implement `decode_decorator()`

* fix exec invocation

* no else blk special case

* add procedure roots comment

* implement forgotten `todo!()`

* `serialize_deserialize_all_nodes` test

* `decode_operations_and_decorators`: fix bit check

* confirm_assumptions test scaffold

* minor  adjustments

* Introduce `StringTableBuilder`

* naming

* test confirm_operation_and_decorator_structure

* remove TODOP

* remove unused `MastNode::new_dyncall()`

* Remove `Error` type

* add TODOP

* complete test `serialize_deserialize_all_nodes`

* check digest on deserialization

* remove TODOP

* safely decode mast node ids

* use method syntax in `MastNodeType` decoding

* TODOPs

* rewrite <= expression

* new `MastNodeType`

* implement `Deserializable` for `MastNodeType`

* migrate tests to new

* Use new MastNodeType

* rename string_table_builder_ module

* implement `BasicBlockDataBuilder`

* add TODOP

* BasicBlockDataDecoder

* use `BasicBlockDataDecoder`

* add headers

* add `MastNodeInfo` method

* return `Result` instead of `Option`

* Remove TODOP

* docs

* chore: add section separators and fix typos

* refactor: change type of the error code of u32assert2 from Felt to u32 (#1382)

* impl `Serializable` for `Operation`

* impl Deserializable for `Operation`

* `StringTableBuilder`: switch to using blake 3

* `EncodedDecoratorVariant`: moved discriminant bit logic to `discriminant()` method

* Remove basic block offset

* Cargo: don't specify patch versions

* make deserialization more efficient

* num-traits and num-derive: set default-features false

* Remove `OperationData`

* `StringRef`: move string length to data buffer

* store offset in block

* Use `source.read_u32/u64()`

* Update `MastNodeInfo` docstring

* rename arguments in `encode_u32_pair`

* Use basic block offset in deserialization

* `BasicBlockDataDecoder`: use `ByteReader::read_u16/32()` methods

* `StringTableBuilder`: fix comment

* Remove `StringRef` in favor of `DataOffset`

* cleanup `MastNodeType` serialization

* derive `Copy` for `MastNodeType`

* `MastNodeType` tests

* add `MastNodeType` tests

* use assert

* fix asserts

* changelog

* fix docs

---------

Co-authored-by: Bobbin Threadbare <[email protected]>
Co-authored-by: Andrey Khmuro <[email protected]>
  • Loading branch information
3 people authored Jul 17, 2024
1 parent 56ccf86 commit b433b57
Show file tree
Hide file tree
Showing 20 changed files with 1,889 additions and 375 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
- Relaxed the parser to allow one branch of an `if.(true|false)` to be empty
- Added support for immediate values for `u32and`, `u32or`, `u32xor` and `u32not` bitwise instructions (#1362).
- Optimized `std::sys::truncate_stuck` procedure (#1384).
- Add serialization/deserialization for `MastForest` (#1370)

#### Changed

Expand Down
2 changes: 1 addition & 1 deletion air/src/constraints/stack/op_flags/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -840,7 +840,7 @@ impl<E: FieldElement> OpFlags<E> {
/// Operation Flag of U32ASSERT2 operation.
#[inline(always)]
pub fn u32assert2(&self) -> E {
self.degree6_op_flags[get_op_index(Operation::U32assert2(ZERO).op_code())]
self.degree6_op_flags[get_op_index(Operation::U32assert2(0).op_code())]
}

/// Operation Flag of U32ADD3 operation.
Expand Down
12 changes: 6 additions & 6 deletions assembly/src/assembler/instruction/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -119,17 +119,17 @@ impl Assembler {
// ----- u32 manipulation -------------------------------------------------------------
Instruction::U32Test => span_builder.push_ops([Dup0, U32split, Swap, Drop, Eqz]),
Instruction::U32TestW => u32_ops::u32testw(span_builder),
Instruction::U32Assert => span_builder.push_ops([Pad, U32assert2(ZERO), Drop]),
Instruction::U32Assert => span_builder.push_ops([Pad, U32assert2(0), Drop]),
Instruction::U32AssertWithError(err_code) => {
span_builder.push_ops([Pad, U32assert2(Felt::from(err_code.expect_value())), Drop])
span_builder.push_ops([Pad, U32assert2(err_code.expect_value()), Drop])
}
Instruction::U32Assert2 => span_builder.push_op(U32assert2(ZERO)),
Instruction::U32Assert2 => span_builder.push_op(U32assert2(0)),
Instruction::U32Assert2WithError(err_code) => {
span_builder.push_op(U32assert2(Felt::from(err_code.expect_value())))
span_builder.push_op(U32assert2(err_code.expect_value()))
}
Instruction::U32AssertW => u32_ops::u32assertw(span_builder, ZERO),
Instruction::U32AssertW => u32_ops::u32assertw(span_builder, 0),
Instruction::U32AssertWWithError(err_code) => {
u32_ops::u32assertw(span_builder, Felt::from(err_code.expect_value()))
u32_ops::u32assertw(span_builder, err_code.expect_value())
}

Instruction::U32Cast => span_builder.push_ops([U32split, Drop]),
Expand Down
5 changes: 2 additions & 3 deletions assembly/src/assembler/instruction/u32_ops.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ use crate::{
use vm_core::{
AdviceInjector, Felt,
Operation::{self, *},
ZERO,
};

/// This enum is intended to determine the mode of operation passed to the parsing function
Expand Down Expand Up @@ -45,7 +44,7 @@ pub fn u32testw(span_builder: &mut BasicBlockBuilder) {
///
/// Implemented by executing `U32ASSERT2` on each pair of elements in the word.
/// Total of 6 VM cycles.
pub fn u32assertw(span_builder: &mut BasicBlockBuilder, err_code: Felt) {
pub fn u32assertw(span_builder: &mut BasicBlockBuilder, err_code: u32) {
#[rustfmt::skip]
let ops = [
// Test the first and the second elements
Expand Down Expand Up @@ -171,7 +170,7 @@ pub fn u32not(span_builder: &mut BasicBlockBuilder) {
let ops = [
// Perform the operation
Push(Felt::from(u32::MAX)),
U32assert2(ZERO),
U32assert2(0),
Swap,
U32sub,

Expand Down
11 changes: 3 additions & 8 deletions core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,14 @@ doctest = false

[features]
default = ["std"]
std = [
"miden-crypto/std",
"miden-formatting/std",
"math/std",
"winter-utils/std",
"thiserror/std",
]
std = ["miden-crypto/std", "miden-formatting/std", "math/std", "winter-utils/std", "thiserror/std"]

[dependencies]
math = { package = "winter-math", version = "0.9", default-features = false }
#miden-crypto = { version = "0.9", default-features = false }
miden-crypto = { git = "https://github.com/0xPolygonMiden/crypto", branch = "next", default-features = false }
miden-formatting = { version = "0.1", default-features = false }
num-derive = { version = "0.4", default-features = false }
num-traits = { version = "0.2", default-features = false }
thiserror = { version = "1.0", git = "https://github.com/bitwalker/thiserror", branch = "no-std", default-features = false }
winter-utils = { package = "winter-utils", version = "0.9", default-features = false }

Expand Down
47 changes: 44 additions & 3 deletions core/src/mast/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,11 @@ use miden_crypto::hash::rpo::RpoDigest;
mod node;
pub use node::{
get_span_op_group_count, BasicBlockNode, CallNode, DynNode, ExternalNode, JoinNode, LoopNode,
MastNode, OpBatch, SplitNode, OP_BATCH_SIZE, OP_GROUP_SIZE,
MastNode, OpBatch, OperationOrDecorator, SplitNode, OP_BATCH_SIZE, OP_GROUP_SIZE,
};
use winter_utils::{ByteReader, ByteWriter, Deserializable, DeserializationError, Serializable};

mod serialization;

#[cfg(test)]
mod tests;
Expand All @@ -18,6 +21,9 @@ pub trait MerkleTreeNode {
fn to_display<'a>(&'a self, mast_forest: &'a MastForest) -> impl fmt::Display + 'a;
}

// MAST NODE ID
// ================================================================================================

/// An opaque handle to a [`MastNode`] in some [`MastForest`]. It is the responsibility of the user
/// to use a given [`MastNodeId`] with the corresponding [`MastForest`].
///
Expand All @@ -27,14 +33,49 @@ pub trait MerkleTreeNode {
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct MastNodeId(u32);

impl MastNodeId {
/// Returns a new `MastNodeId` with the provided inner value, or an error if the provided
/// `value` is greater than the number of nodes in the forest.
///
/// For use in deserialization.
pub fn from_u32_safe(
value: u32,
mast_forest: &MastForest,
) -> Result<Self, DeserializationError> {
if (value as usize) < mast_forest.nodes.len() {
Ok(Self(value))
} else {
Err(DeserializationError::InvalidValue(format!(
"Invalid deserialized MAST node ID '{}', but only {} nodes in the forest",
value,
mast_forest.nodes.len(),
)))
}
}
}

impl fmt::Display for MastNodeId {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "MastNodeId({})", self.0)
}
}

impl Serializable for MastNodeId {
fn write_into<W: ByteWriter>(&self, target: &mut W) {
self.0.write_into(target)
}
}

impl Deserializable for MastNodeId {
fn read_from<R: ByteReader>(source: &mut R) -> Result<Self, DeserializationError> {
let inner = source.read_u32()?;

Ok(Self(inner))
}
}

// MAST FOREST
// ===============================================================================================
// ================================================================================================

/// Represents one or more procedures, represented as a collection of [`MastNode`]s.
///
Expand Down Expand Up @@ -94,7 +135,7 @@ impl MastForest {
/// Returns the [`MastNode`] associated with the provided [`MastNodeId`] if valid, or else
/// `None`.
///
/// This is the faillible version of indexing (e.g. `mast_forest[node_id]`).
/// This is the failable version of indexing (e.g. `mast_forest[node_id]`).
#[inline(always)]
pub fn get_node_by_id(&self, node_id: MastNodeId) -> Option<&MastNode> {
let idx = node_id.0 as usize;
Expand Down
Loading

0 comments on commit b433b57

Please sign in to comment.