Skip to content

Commit

Permalink
feat: serialize fields as json (#165)
Browse files Browse the repository at this point in the history
* split input notes

* make enum for notes tables

* rename recipients variable and add more NoteTable uses

* run fmt

* change to_string for to_hex

* correct nullable fields and reorder fields

* remove unnecesarry note filter

* remove unnecessary commit_height field

* move related rows into metadata row for notes

* remove dbg statements

* switch to_string for to_hex

* Revert "move related rows into metadata row for notes"

This reverts commit bf46c0a.

* restore filtering by pending notes

* store grouped json fields

* add json fields with serialization/deserialization

I still have pending polishing the changes (abstracting some deserialization and removing debugs, etc.) and we should also add some kind of validation as well as having the proper NULL / NOT NULL columns (now everything is nullable)

* remove dbg calls and set nullable/non-nullable cols accordingly

* apply clippy suggestion

* add helper functions to work with json values returned as strings

* add documentation for handling json with sqlite

* fix incorrect function use to fetch output notes

* ignore doc tests

* add json validity constraints

* use structs to represent json columns

* remove dbg statements

* use proper serialization

* use existing NoteMetadata instead of NoteRecordMetadata

* also remove NoteRecordInclusionProof and use NoteInclusionProof instead
  • Loading branch information
mFragaBA authored Mar 13, 2024
1 parent 29833cc commit 3c8316f
Show file tree
Hide file tree
Showing 6 changed files with 322 additions and 153 deletions.
6 changes: 6 additions & 0 deletions src/errors.rs
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,12 @@ impl From<MmrError> for StoreError {
}
}

impl From<NoteError> for StoreError {
fn from(value: NoteError) -> Self {
StoreError::NoteInclusionProofError(value)
}
}

impl From<TransactionScriptError> for StoreError {
fn from(value: TransactionScriptError) -> Self {
StoreError::TransactionScriptError(value)
Expand Down
37 changes: 35 additions & 2 deletions src/store/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@ use miden_objects::{
utils::collections::BTreeMap,
BlockHeader, Digest, Word,
};

use miden_tx::utils::{ByteReader, ByteWriter, Deserializable, DeserializationError, Serializable};
use serde::{Deserialize, Serialize};

pub mod data_store;
pub mod sqlite_store;
Expand Down Expand Up @@ -315,8 +317,8 @@ impl Deserializable for InputNoteRecord {
fn read_from<R: ByteReader>(
source: &mut R,
) -> std::prelude::v1::Result<Self, DeserializationError> {
let note: Note = source.read()?;
let proof: Option<NoteInclusionProof> = source.read()?;
let note = Note::read_from(source)?;
let proof = Option::<NoteInclusionProof>::read_from(source)?;
Ok(InputNoteRecord::new(note, proof))
}
}
Expand Down Expand Up @@ -354,6 +356,37 @@ impl TryInto<InputNote> for InputNoteRecord {
}
}

#[derive(Serialize, Deserialize)]
struct NoteRecordDetails {
nullifier: String,
script: Vec<u8>,
inputs: Vec<u8>,
serial_num: Word,
}

impl NoteRecordDetails {
fn new(nullifier: String, script: Vec<u8>, inputs: Vec<u8>, serial_num: Word) -> Self {
Self {
nullifier,
script,
inputs,
serial_num,
}
}

fn script(&self) -> &Vec<u8> {
&self.script
}

fn inputs(&self) -> &Vec<u8> {
&self.inputs
}

fn serial_num(&self) -> &Word {
&self.serial_num
}
}

// CHAIN MMR NODE FILTER
// ================================================================================================

Expand Down
60 changes: 59 additions & 1 deletion src/store/sqlite_store/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,65 @@ mod transactions;

// SQLITE STORE
// ================================================================================================

///
/// Represents a connection with an sqlite database
///
///
/// Current table definitions can be found at `store.sql` migration file. One particular column
/// type used is JSON, for which you can look more info at [sqlite's official documentation](https://www.sqlite.org/json1.html).
/// In the case of json, some caveats must be taken:
///
/// - To insert json values you must use sqlite's `json` function in the query alongside named
/// parameters, and the provided parameter must be a valid json. That is:
///
/// ```sql
/// INSERT INTO SOME_TABLE
/// (some_field)
/// VALUES (json(:some_field))")
/// ```
///
/// ```ignore
/// let metadata = format!(r#"{{"some_inner_field": {some_field}, "some_other_inner_field": {some_other_field}}}"#);
/// ```
///
/// (Using raw string literals for the jsons is encouraged if possible)
///
/// - To get data from any of the json fields you can use the `json_extract` function (in some
/// cases you'll need to do some explicit type casting to help rusqlite figure out types):
///
/// ```sql
/// SELECT CAST(json_extract(some_json_col, '$.some_json_field') AS TEXT) from some_table
/// ```
///
/// - For some datatypes you'll need to do some manual serialization/deserialization. For example,
/// suppose one of your json fields is an array of digests. Then you'll need to
/// - Create the json with an array of strings representing the digests:
///
/// ```ignore
/// let some_array_field = some_array
/// .into_iter()
/// .map(array_elem_to_string)
/// .collect::<Vec<_>>()
/// .join(",");
///
/// Some(format!(
/// r#"{{
/// "some_array_field": [{some_array_field}]
/// }}"#
/// )),
/// ```
///
/// - When deserializing, handling the extra symbols (`[`, `]`, `,`, `"`). For that you can use
/// the `parse_json_array` function:
///
/// ```ignore
/// let some_array = parse_json_array(some_array_field)
/// .into_iter()
/// .map(parse_json_byte_str)
/// .collect::<Result<Vec<u8>, _>>()?;
/// ```
/// - Thus, if needed you can create a struct representing the json values and use serde_json to
/// simplify all of the serialization/deserialization logic
pub struct SqliteStore {
pub(crate) db: Connection,
}
Expand Down
Loading

0 comments on commit 3c8316f

Please sign in to comment.