Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MastForest serialization #1370

Merged
merged 130 commits into from
Jul 17, 2024
Merged
Show file tree
Hide file tree
Changes from 126 commits
Commits
Show all changes
130 commits
Select commit Hold shift + click to select a range
a94d52d
Introduce `ExternalNode`
plafer Jun 20, 2024
f8ad339
Replace `Assembler.node_id_by_digest` map
plafer Jun 20, 2024
06f421b
add TODOP
plafer Jun 20, 2024
bcf2a9b
Add `Host::get_mast_forest`
plafer Jun 20, 2024
980713d
Move kernel and entrypoint out of `MastForest`
plafer Jun 21, 2024
d2bbcd2
Add Host::get_mast_forest
plafer Jun 21, 2024
32e757e
Remove ProgramError
plafer Jun 21, 2024
f87217a
docs
plafer Jun 21, 2024
71f35f1
cleanup Program constructors
plafer Jun 21, 2024
a39f74f
fix docs
plafer Jun 21, 2024
f7e98af
Make `Program.kernel` an `Arc`
plafer Jun 21, 2024
b5538c5
fix executable
plafer Jun 21, 2024
2eab574
Merge remote-tracking branch 'origin/next' into plafer-object-store
plafer Jun 21, 2024
188651b
invoke_mast_root: fix external node creation logic
plafer Jun 23, 2024
b15263c
add failing test
plafer Jun 23, 2024
a94a095
don't make root in `combine_mast_node_ids` and `compile_body`
plafer Jun 23, 2024
7af0bfa
fix External docs
plafer Jun 23, 2024
a6fcf47
fmt
plafer Jun 23, 2024
c66db6f
fix `entrypoint` doc
plafer Jun 23, 2024
572fc7e
Rename `Program::new_with_kernel()`
plafer Jun 23, 2024
08ce2c7
Document `MastForestStore` and `MemMastForestStore`
plafer Jun 23, 2024
50e01e9
fix syscall
plafer Jun 23, 2024
071ab54
execute_* functions: use `MastForest`
plafer Jun 23, 2024
49de40d
`Program`: Remove `Arc` around kernel
plafer Jun 23, 2024
c28c876
remove `Arc` around `MastForest` in `Program`
plafer Jun 23, 2024
78b2b16
Return error on malformed host
plafer Jun 23, 2024
4883b44
Simplify `DefaultHost`
plafer Jun 23, 2024
155a798
`MastForest::add_node()`: add docs
plafer Jun 23, 2024
bc6d13e
fmt
plafer Jun 23, 2024
be24320
add failing `duplicate_procedure()` test
plafer Jun 23, 2024
32aedd6
Introduce `MastForestBuilder`
plafer Jun 23, 2024
088de82
Rename `mod tests` -> `testing`
plafer Jun 25, 2024
9d48fda
add `duplicate_node()` test
plafer Jun 25, 2024
6c62d9b
changelog
plafer Jun 25, 2024
039bba0
Program: use `assert!()` instead of `debug_assert!()`
plafer Jun 26, 2024
9c9e171
`MastForest::make_root()`: add assert
plafer Jun 26, 2024
8e4dc5e
Merge remote-tracking branch 'origin/next' into plafer-object-store
plafer Jun 26, 2024
c34e985
fmt
plafer Jun 26, 2024
c1b2693
Serialization for `MastNodeId`
plafer Jun 26, 2024
efc24fd
serialization for MastNode variants except basic block
plafer Jun 26, 2024
861d0d5
MastForest serialization scaffolding
plafer Jun 27, 2024
6173790
define `MastNodeType` constructor from `MastNode`
plafer Jun 27, 2024
e018cc8
test join serialization of MastNodeType
plafer Jun 27, 2024
6671afe
`MastNodeType` serialization of split
plafer Jun 27, 2024
babdd0c
Revert "serialization for MastNode variants except basic block"
plafer Jun 27, 2024
cd06527
add TODOP
plafer Jun 27, 2024
2dd4829
impl Deserializable for `MastForest` (scaffold)
plafer Jul 1, 2024
4be1401
mast_node_to_info() scaffold
plafer Jul 1, 2024
2695062
try_info_to_mast_node scaffold
plafer Jul 1, 2024
ef0a881
Rename `EncodedMastNodeType`
plafer Jul 1, 2024
dd89461
add info module
plafer Jul 1, 2024
91009b0
encode operations into `data` field
plafer Jul 1, 2024
d5ed108
decode operations
plafer Jul 1, 2024
0cf49ff
implement `BasicBlockNode::num_operations_and_decorators()`
plafer Jul 2, 2024
64c36ec
OperationOrDecoratorIterator
plafer Jul 2, 2024
794ebbb
basic block node: move tests in new file
plafer Jul 2, 2024
49673ae
operation_or_decorator_iterator test
plafer Jul 2, 2024
a5c324b
Implement `Operation::with_opcode_and_data()`
plafer Jul 2, 2024
c04bc90
encode decorators
plafer Jul 2, 2024
27e2783
implement `decode_decorator()`
plafer Jul 2, 2024
25fe82f
fix exec invocation
plafer Jul 2, 2024
36ecdd7
Merge remote-tracking branch 'origin/next' into plafer-object-store
plafer Jul 2, 2024
3c26bd6
no else blk special case
plafer Jul 2, 2024
34c2f7f
add procedure roots comment
plafer Jul 2, 2024
97d3de8
Merge remote-tracking branch 'origin/next' into plafer-object-store
plafer Jul 3, 2024
3166dc2
Merge branch 'plafer-object-store' into plafer-mast-forest-serialization
plafer Jul 3, 2024
ffc7c78
implement forgotten `todo!()`
plafer Jul 3, 2024
5504157
`serialize_deserialize_all_nodes` test
plafer Jul 3, 2024
93c4fca
`decode_operations_and_decorators`: fix bit check
plafer Jul 3, 2024
da14984
confirm_assumptions test scaffold
plafer Jul 3, 2024
4870b32
minor adjustments
plafer Jul 3, 2024
072b0b9
Introduce `StringTableBuilder`
plafer Jul 3, 2024
a04836c
Merge remote-tracking branch 'origin/next' into plafer-mast-forest-se…
plafer Jul 3, 2024
48aec6a
naming
plafer Jul 3, 2024
a4ef4b1
test confirm_operation_and_decorator_structure
plafer Jul 3, 2024
ca0e7fe
remove TODOP
plafer Jul 3, 2024
78e35fc
remove unused `MastNode::new_dyncall()`
plafer Jul 3, 2024
411f9f3
Remove `Error` type
plafer Jul 3, 2024
624984c
add TODOP
plafer Jul 3, 2024
858582a
complete test `serialize_deserialize_all_nodes`
plafer Jul 3, 2024
4e5efd3
check digest on deserialization
plafer Jul 3, 2024
3f522b8
remove TODOP
plafer Jul 3, 2024
161578d
safely decode mast node ids
plafer Jul 3, 2024
2ab1cf9
use method syntax in `MastNodeType` decoding
plafer Jul 3, 2024
c8cfa8e
TODOPs
plafer Jul 3, 2024
a360959
rewrite <= expression
plafer Jul 3, 2024
c8ba463
new `MastNodeType`
plafer Jul 3, 2024
ad18580
implement `Deserializable` for `MastNodeType`
plafer Jul 3, 2024
996498f
migrate tests to new
plafer Jul 3, 2024
e60bfc2
Use new MastNodeType
plafer Jul 3, 2024
578bda9
rename string_table_builder_ module
plafer Jul 3, 2024
d25671c
implement `BasicBlockDataBuilder`
plafer Jul 3, 2024
99f4544
add TODOP
plafer Jul 3, 2024
0d20629
BasicBlockDataDecoder
plafer Jul 3, 2024
b66f81b
use `BasicBlockDataDecoder`
plafer Jul 3, 2024
f9a3a0b
add headers
plafer Jul 3, 2024
df621c5
add `MastNodeInfo` method
plafer Jul 3, 2024
2afd588
return `Result` instead of `Option`
plafer Jul 3, 2024
9106a6f
Remove TODOP
plafer Jul 3, 2024
fe8b7a7
docs
plafer Jul 3, 2024
5869525
chore: add section separators and fix typos
bobbinth Jul 7, 2024
8cb3462
refactor: change type of the error code of u32assert2 from Felt to u3…
Fumuran Jul 10, 2024
5ec4826
impl `Serializable` for `Operation`
plafer Jul 10, 2024
d4a50e9
impl Deserializable for `Operation`
plafer Jul 10, 2024
3ce71b9
`StringTableBuilder`: switch to using blake 3
plafer Jul 10, 2024
db33dc7
`EncodedDecoratorVariant`: moved discriminant bit logic to `discrimin…
plafer Jul 10, 2024
10f02a6
Remove basic block offset
plafer Jul 10, 2024
421518d
Cargo: don't specify patch versions
plafer Jul 10, 2024
cf100c5
make deserialization more efficient
plafer Jul 10, 2024
f50073d
num-traits and num-derive: set default-features false
plafer Jul 10, 2024
aacb9e9
Merge branch 'next' into plafer-mast-forest-serialization
plafer Jul 11, 2024
956aac1
Remove `OperationData`
plafer Jul 11, 2024
9fe4e0e
`StringRef`: move string length to data buffer
plafer Jul 11, 2024
c525828
store offset in block
plafer Jul 11, 2024
2b332e1
Use `source.read_u32/u64()`
plafer Jul 12, 2024
9ca910a
Update `MastNodeInfo` docstring
plafer Jul 12, 2024
5c6f287
rename arguments in `encode_u32_pair`
plafer Jul 12, 2024
58df61e
Use basic block offset in deserialization
plafer Jul 12, 2024
1cdc419
`BasicBlockDataDecoder`: use `ByteReader::read_u16/32()` methods
plafer Jul 12, 2024
f45d0c5
`StringTableBuilder`: fix comment
plafer Jul 12, 2024
43cadeb
Remove `StringRef` in favor of `DataOffset`
plafer Jul 12, 2024
7dec428
cleanup `MastNodeType` serialization
plafer Jul 12, 2024
c081b00
derive `Copy` for `MastNodeType`
plafer Jul 12, 2024
c32ef22
`MastNodeType` tests
plafer Jul 12, 2024
c7ee9c8
add `MastNodeType` tests
plafer Jul 12, 2024
f9d2e59
use assert
plafer Jul 12, 2024
781fc73
fix asserts
plafer Jul 12, 2024
04e6b0a
Merge branch 'next' into plafer-mast-forest-serialization
plafer Jul 17, 2024
0a9cfad
changelog
plafer Jul 17, 2024
40cbbb9
fix docs
plafer Jul 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion air/src/constraints/stack/op_flags/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -840,7 +840,7 @@ impl<E: FieldElement> OpFlags<E> {
/// Operation Flag of U32ASSERT2 operation.
#[inline(always)]
pub fn u32assert2(&self) -> E {
self.degree6_op_flags[get_op_index(Operation::U32assert2(ZERO).op_code())]
self.degree6_op_flags[get_op_index(Operation::U32assert2(0).op_code())]
}

/// Operation Flag of U32ADD3 operation.
Expand Down
12 changes: 6 additions & 6 deletions assembly/src/assembler/instruction/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -119,17 +119,17 @@ impl Assembler {
// ----- u32 manipulation -------------------------------------------------------------
Instruction::U32Test => span_builder.push_ops([Dup0, U32split, Swap, Drop, Eqz]),
Instruction::U32TestW => u32_ops::u32testw(span_builder),
Instruction::U32Assert => span_builder.push_ops([Pad, U32assert2(ZERO), Drop]),
Instruction::U32Assert => span_builder.push_ops([Pad, U32assert2(0), Drop]),
Instruction::U32AssertWithError(err_code) => {
span_builder.push_ops([Pad, U32assert2(Felt::from(err_code.expect_value())), Drop])
span_builder.push_ops([Pad, U32assert2(err_code.expect_value()), Drop])
}
Instruction::U32Assert2 => span_builder.push_op(U32assert2(ZERO)),
Instruction::U32Assert2 => span_builder.push_op(U32assert2(0)),
Instruction::U32Assert2WithError(err_code) => {
span_builder.push_op(U32assert2(Felt::from(err_code.expect_value())))
span_builder.push_op(U32assert2(err_code.expect_value()))
}
Instruction::U32AssertW => u32_ops::u32assertw(span_builder, ZERO),
Instruction::U32AssertW => u32_ops::u32assertw(span_builder, 0),
Instruction::U32AssertWWithError(err_code) => {
u32_ops::u32assertw(span_builder, Felt::from(err_code.expect_value()))
u32_ops::u32assertw(span_builder, err_code.expect_value())
}

Instruction::U32Cast => span_builder.push_ops([U32split, Drop]),
Expand Down
5 changes: 2 additions & 3 deletions assembly/src/assembler/instruction/u32_ops.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ use crate::{
use vm_core::{
AdviceInjector, Felt,
Operation::{self, *},
ZERO,
};

/// This enum is intended to determine the mode of operation passed to the parsing function
Expand Down Expand Up @@ -45,7 +44,7 @@ pub fn u32testw(span_builder: &mut BasicBlockBuilder) {
///
/// Implemented by executing `U32ASSERT2` on each pair of elements in the word.
/// Total of 6 VM cycles.
pub fn u32assertw(span_builder: &mut BasicBlockBuilder, err_code: Felt) {
pub fn u32assertw(span_builder: &mut BasicBlockBuilder, err_code: u32) {
#[rustfmt::skip]
let ops = [
// Test the first and the second elements
Expand Down Expand Up @@ -171,7 +170,7 @@ pub fn u32not(span_builder: &mut BasicBlockBuilder) {
let ops = [
// Perform the operation
Push(Felt::from(u32::MAX)),
U32assert2(ZERO),
U32assert2(0),
Swap,
U32sub,

Expand Down
11 changes: 3 additions & 8 deletions core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,14 @@ doctest = false

[features]
default = ["std"]
std = [
"miden-crypto/std",
"miden-formatting/std",
"math/std",
"winter-utils/std",
"thiserror/std",
]
std = ["miden-crypto/std", "miden-formatting/std", "math/std", "winter-utils/std", "thiserror/std"]

[dependencies]
math = { package = "winter-math", version = "0.9", default-features = false }
#miden-crypto = { version = "0.9", default-features = false }
miden-crypto = { git = "https://github.com/0xPolygonMiden/crypto", branch = "next", default-features = false }
miden-formatting = { version = "0.1", default-features = false }
num-derive = { version = "0.4", default-features = false }
num-traits = { version = "0.2", default-features = false }
thiserror = { version = "1.0", git = "https://github.com/bitwalker/thiserror", branch = "no-std", default-features = false }
winter-utils = { package = "winter-utils", version = "0.9", default-features = false }

Expand Down
47 changes: 44 additions & 3 deletions core/src/mast/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,11 @@ use miden_crypto::hash::rpo::RpoDigest;
mod node;
pub use node::{
get_span_op_group_count, BasicBlockNode, CallNode, DynNode, JoinNode, LoopNode, MastNode,
OpBatch, SplitNode, OP_BATCH_SIZE, OP_GROUP_SIZE,
OpBatch, OperationOrDecorator, SplitNode, OP_BATCH_SIZE, OP_GROUP_SIZE,
};
use winter_utils::{ByteReader, ByteWriter, Deserializable, DeserializationError, Serializable};

mod serialization;

#[cfg(test)]
mod tests;
Expand All @@ -18,6 +21,9 @@ pub trait MerkleTreeNode {
fn to_display<'a>(&'a self, mast_forest: &'a MastForest) -> impl fmt::Display + 'a;
}

// MAST NODE ID
// ================================================================================================

/// An opaque handle to a [`MastNode`] in some [`MastForest`]. It is the responsibility of the user
/// to use a given [`MastNodeId`] with the corresponding [`MastForest`].
///
Expand All @@ -27,14 +33,49 @@ pub trait MerkleTreeNode {
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct MastNodeId(u32);

impl MastNodeId {
/// Returns a new `MastNodeId` with the provided inner value, or an error if the provided
/// `value` is greater than the number of nodes in the forest.
///
/// For use in deserialization.
pub fn from_u32_safe(
value: u32,
mast_forest: &MastForest,
) -> Result<Self, DeserializationError> {
if (value as usize) < mast_forest.nodes.len() {
Ok(Self(value))
} else {
Err(DeserializationError::InvalidValue(format!(
"Invalid deserialized MAST node ID '{}', but only {} nodes in the forest",
value,
mast_forest.nodes.len(),
)))
}
}
}

impl fmt::Display for MastNodeId {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "MastNodeId({})", self.0)
}
}

impl Serializable for MastNodeId {
fn write_into<W: ByteWriter>(&self, target: &mut W) {
self.0.write_into(target)
}
}

impl Deserializable for MastNodeId {
fn read_from<R: ByteReader>(source: &mut R) -> Result<Self, DeserializationError> {
let inner = source.read_u32()?;

Ok(Self(inner))
}
}

// MAST FOREST
// ===============================================================================================
// ================================================================================================

/// Represents one or more procedures, represented as a collection of [`MastNode`]s.
///
Expand Down Expand Up @@ -94,7 +135,7 @@ impl MastForest {
/// Returns the [`MastNode`] associated with the provided [`MastNodeId`] if valid, or else
/// `None`.
///
/// This is the faillible version of indexing (e.g. `mast_forest[node_id]`).
/// This is the failable version of indexing (e.g. `mast_forest[node_id]`).
#[inline(always)]
pub fn get_node_by_id(&self, node_id: MastNodeId) -> Option<&MastNode> {
let idx = node_id.0 as usize;
Expand Down
Loading
Loading