Added ledger item management

proposed/0000-structured-data-timestamping.md

@@ -54,37 +54,58 @@ During the validation consensus a manager has only to check that its own date is
 It is stored in 2 new fields. The i64 type represents the sec part of a struct Timespec
 generated from an UTC time (a number of seconds since the beginning of the epoch).
 
+These fields are embedded in an enum that indicates whether or not the SD is a ledger item.
+This enum specify the delete behaviour so that a ledger item cannot be recreated with the
+same name and the same version but a different content.
+
 ```rust
-    min_date: i64,
-    max_date: i64,
+/// Indicate if SD is a ledger item.
+///
+/// A ledger item is a SD that is not erased from the network when it is deleted.
+/// Instead a new version is stored with an empty payload and its version is incremented by 1
+/// (equivalent to a Post command).
+/// Optionally, a timestamp can be specified.
+#[derive(Hash, Eq, PartialEq, PartialOrd, Ord, Clone, RustcDecodable, RustcEncodable)]
+pub enum LedgerItem {
+    /// SD is not a ledger item
+    None,
+    /// SD is a ledger item but has no timestamp
+    NoTimestamp,
+    /// SD is a ledger item and has a timestamp
+    WithTimestamp {
+        /// Lower bond of timestamp
+        min_date: i64,
+        /// Upper bond of timestamp
+        max_date: i64,
+    }
+}
 ```
 
-## Writing of default values in the existing constructor (StructuredData::new)
+## Writing of a default value in the existing constructor (StructuredData::new)
 
-The values insure that the behavior of classic SD's remains unchanged with no date validation:
+This value insures that the behavior of classic SD's remains unchanged (they are not ledger items
+and have no date validation).
 
 ```rust
-            min_date : 0i64,
-            max_date : i64::max_value(),
+            ledger_item: LedgerItem::None,
 ```
 
 ## Addition of a new constructor
 
-A new constructor (StructuredData::with_dates) with 2 supplementary arguments
-specifies a range of dates for the timestamping:
+A new constructor (StructuredData::with_ledger_item) with a supplementary argument that specifies
+if SD is ledger item with an optional range of dates for timestamping:
 
 ```rust
-    /// Constructor with min/max dates.
-    /// StructuredData::new cannot simply be called and then dates added because dates must be part of signed data.
-    pub fn with_dates(type_tag: u64,
-               identifier: ::NameType,
+    /// Constructor with ledger item.
+    /// StructuredData::new cannot simply be called and then ledger item added because ledger item must be part of signed data.
+    pub fn with_ledger_item(type_tag: u64,
+               identifier: XorName,
                version: u64,
                data: Vec<u8>,
-               current_owner_keys: Vec<::sodiumoxide::crypto::sign::PublicKey>,
-               previous_owner_keys: Vec<::sodiumoxide::crypto::sign::PublicKey>,
-               signing_key: Option<&::sodiumoxide::crypto::sign::SecretKey>,
-               min_date: i64,
-               max_date: i64)
+               current_owner_keys: Vec<PublicKey>,
+               previous_owner_keys: Vec<PublicKey>,
+               signing_key: Option<&SecretKey>,
+               ledger_item: LedgerItem)
                -> Result<StructuredData, ::error::RoutingError> {
 
         let mut structured_data = StructuredData {
@@ -95,8 +116,7 @@ specifies a range of dates for the timestamping:
             version: version,
             current_owner_keys: current_owner_keys,
             previous_owner_signatures: vec![],
-            min_date : min_date,
-            max_date : max_date,
+            ledger_item: ledger_item,
         };
 
         if let Some(key) = signing_key {
@@ -106,14 +126,13 @@ specifies a range of dates for the timestamping:
     }
 ```
 
-## Inclusion of the new fields in the replace function
+## Inclusion of the new field in the replace function
 
 This is done in replace_with_other function.
-The date fields must be replaced when an SD is updated.
+The ledger item must be replaced when an SD is updated.
 
 ```rust
-        self.min_date = other.min_date;
-        self.max_date = other.max_date;
+        self.ledger_item = other.ledger_item;
 ```
 
 ## Validation function
@@ -124,72 +143,114 @@ It checks that current date is within the specified range.
     /// Validate date. An error is generated when current date is not in the range [min_date .. max_date]
     pub fn validate_date(&self) -> Result<(), ::error::RoutingError> {
         use time;
-        // Don't compute utc time for standard SD (with default values for dates)
-        if self.min_date <= 0 && self.max_date == i64::max_value() {
-            return Ok(())
-        }
-        let now_utc = time::now_utc().to_timespec().sec;
-        if now_utc < self.min_date || now_utc > self.max_date {
-            Err(::error::RoutingError::OutOfRangeDate)
-        }
-        else {
-            Ok(())
+        match self.ledger_item {
+            LedgerItem::WithTimestamp { min_date, max_date } => {
+                let now_utc = time::now_utc().to_timespec().sec;
+                if now_utc < min_date || now_utc > max_date {
+                    Err(::error::RoutingError::OutOfRangeDate)
+                }
+                else {
+                    Ok(())
+                }
+            },
+            // Don't validate date of non ledger item or ledger item without dates
+            _ => Ok(())
         }
     }
 ```
 
-## Inclusion of the new fields in the signed part of StructuredData
+## Validation against successor
+
+To control that a ledger item is not replaced by a non ledger item the following code is added to
+validate_self_against_successor function:
+
+```rust
+        // Replacing a ledger item by a non ledger item is forbidden because this would allow complete deletion
+        // and then recreation of SD.
+        if other.ledger_item == LedgerItem::None && self.ledger_item != LedgerItem::None {
+            return Err(::error::RoutingError::LedgerItemReplaced);
+        }
+```
+
+## Inclusion of the new field in the signed part of StructuredData
 
 This is done in data_to_sign function.
-The aim is to prove that date fields haven't been tampered with (like the rest of the SD).
+The aim is to prove that ledger item field hasn't been tampered with (like the rest of the SD).
+
+```rust
+            ledger_item: self.ledger_item.clone(),
+```
+
+## Expose LedgerItem type to clients
+
+To facilitate usage of leger items, this type must be exposed at the root level of routing namespace in lib.rs:
 
 ```rust
-        try!(enc.encode(self.min_date.to_string().as_bytes()));
-        try!(enc.encode(self.max_date.to_string().as_bytes()));
+pub use structured_data::{MAX_STRUCTURED_DATA_SIZE_IN_BYTES, StructuredData, LedgerItem};
 ```
 
-## Two public getter functions
+## Public getter functions
+
+Clients can get the timestamp of a SD by calling these functions. If the SD is not a legder item or is a ledger item without timestamp,
+their return values defines a timestamp with a range from beginning of EPOCH to 2^63 - 1 seconds after it
+(which means no timestamp concretely).
 
-Clients can get the timestamp of a SD by calling these functions.
 
 ```rust
-    /// Get the min date
+    /// Get timestamp lower bound
     pub fn get_min_date(&self) -> i64 {
-        self.min_date
+        match self.ledger_item {
+            LedgerItem::WithTimestamp { min_date, .. } => {
+                min_date
+            }
+            // No timestamp
+            _ => 0i64
+        }
     }
 
-    /// Get the max date
+    /// Get timestamp upper bound
     pub fn get_max_date(&self) -> i64 {
-        self.max_date
+        match self.ledger_item {
+            LedgerItem::WithTimestamp { max_date, .. } => {
+                max_date
+            }
+            // No timestamp
+            _ => i64::max_value()
+        }
+    }
+```
+
+Clients can also test is a SD is a ledger item:
+
+```rust
+    /// Test if SD is a ledger item
+    pub fn is_ledger_item(&self) -> bool {
+        self.ledger_item != LedgerItem::None
     }
 ```
 
-## New error code
+## New error codes in RoutingError enumeration
 
 OutOfRangeDate is returned by the validation function when current UTC time is not inside the specified range.
-The code is defined at 3 places:
 
-RoutingError enumeration:
 ```rust
     /// Current date is ouside specified range [min_date .. max_date] of structured data
     OutOfRangeDate,
 ```
 
-Implementation of std::error::Error for RoutingError:
-```rust
-            RoutingError::OutOfRangeDate => "Current date is outside specified range",
-```
+LedgerItemReplaced is returned by validate_self_against_successor function when an attempt is made to change
+a ledger item by a non ledger item. This is forbidden because this would allow real deletion and then recreation
+of SD.
 
-Implementation of std::fmt::Display for RoutingError:
 ```rust
-            RoutingError::OutOfRangeDate =>
-                ::std::fmt::Display::fmt("Current date is outside specified range", formatter),
+    /// Ledger item replaced by non ledger item
+    LedgerItemReplaced,
 ```
 
 ## Usage from clients
 
 Current client programs will behave the same as before with no date validation.
-The new constructor (StructuredData::with_dates) has to be called
+The new constructor (StructuredData::with_ledger_item) has to be called
 to activate the timestamping functionality.
 
 To maximize the chances that its request is valid the client can give a very large range
@@ -214,42 +275,95 @@ So, backdating a document is not possible.
 Date validation cannot be done at later stages, like for example during a churn event.
 This is mitigated by the fact that signatures
 can be checked again at these stages, which will prove that the date fields haven't been tampered with.
-This is the same way that others fields are revalidated (like the data itself).  
+This is the same way that others fields are revalidated (like the data itself).
 
 # Alternatives
 
 Other designs with time synchronization between nodes are a lot of more complex to implement.
-This one is really simple with only 64 new lines of code (not counting test functions).
+This one is really simple with about 100 new lines of code (not counting test functions).
 
-# Unresolved questions
+# Solved questions
 
-## Where to call the date validation function? 
+## Where to call the date validation function?
 
-The date validation can be done in the maid managers or the sd managers or both,
-I would say it should be done at the same place(s) where the signatures of SD content are verified
-(except churn or later events).
+The date validation can be done in the maid managers or the data managers or both,
 
-If done in sd managers the following pieces of code are to be added in safe_vault/src/sd_manager/mod.rs:
+If done in data managers the following pieces of code are to be added in safe_vault/src/personas/data_manager.rs:
 
 - In function handle_put:
 
 ```rust
-        // Validate min/max dates
-        if structured_data.validate_date().is_err() {
-            warn!("Invalid min/max dates for PUT at StructuredDataManager");
-            return ::utils::HANDLED;
+        if let Data::Structured(ref data) = data {
+            // Validate min/max dates
+            if let Err(_) = data.validate_date() {
+                let error = MutationError::InvalidSuccessor;
+                let external_error_indicator = try!(serialisation::serialise(&error));
+                trace!("DM sending PutFailure for data {:?}, invalid timestamp.",
+                       data_id);
+                let _ = self.routing_node
+                    .send_put_failure(dst, src, data_id, external_error_indicator, message_id);
+                return Err(From::from(error));
+            }
         }
 ```
 - In function handle_post:
 
 ```rust
         // Validate min/max dates
-        if new_data.validate_date().is_err() {
-            warn!("Invalid min/max dates for POST at StructuredDataManager");
-            return ::utils::HANDLED;
+        if let Err(_) = new_data.validate_date() {
+            let error = MutationError::InvalidSuccessor;
+            let external_error_indicator = try!(serialisation::serialise(&error));
+            trace!("DM sending post_failure for data {:?}, invalid timestamp.",
+                   data_id);
+            let _ = self.routing_node
+                .send_put_failure(dst, src, data_id, external_error_indicator, message_id);
+            return Err(From::from(error));
         }
 ```
 
+## Where to manage deletion of ledger items?
+
+This is to be done in function handle_delete of safe_vault/src/personas/data_manager.rs. The following code
+should be executed when validation against successor is OK:
+
+```rust
+                if !data.is_ledger_item() {
+                    // Not a ledger item => erase SD from chunk store
+                    if let Ok(()) = self.chunk_store.delete(&data_id) {
+                        self.count_removed_data(&data_id);
+                        trace!("DM deleted {:?}", data.identifier());
+                        info!("{:?}", self);
+                        let _ = self.routing_node.send_delete_success(dst, src, data_id, message_id);
+                        // TODO: Send a refresh message.
+                        return Ok(());
+                    }
+                } else {
+                    // Ledger item => check that data part is empty and store empty SD in chunk store
+                    if new_data.payload_size() == 0 && new_data.validate_date().is_ok() {
+                        let version = new_data.get_version();
+                        if let Err(error) = self.chunk_store.put(&data_id, &Data::Structured(new_data)) {
+                            trace!("DM sending delete_failure for: {:?} with {:?} - {:?}",
+                                   data_id,
+                                   message_id,
+                                   error);
+                            let mutation_error =
+                                MutationError::NetworkOther(format!("Failed to store chunk: {:?}", error));
+                            let post_error = try!(serialisation::serialise(&mutation_error));
+                            return Ok(try!(self.routing_node
+                                .send_delete_failure(dst, src, data_id, post_error, message_id)));
+                        }
+                        trace!("DM updated for: {:?}", data_id);
+                        let _ = self.routing_node.send_delete_success(dst, src, data_id, message_id);
+                        let data_list = vec![(data_id, version)];
+                        let _ = self.send_refresh(Authority::NaeManager(data_id.name()), data_list);
+                        return Ok(())
+                    }
+                }
+```
+
+
+# Unresolved question
+
 ## Can it work if date variability among the nodes is too important?
 
 This solution only works if a consensus majority of nodes (28/32) have a reasonable difference with actual time.