Rename feature

docs/format.rst

@@ -484,56 +484,62 @@ document. Until they are defined in the specification, there is no guarantee tha
 readers will be able to safely interpret new forms of statistics.
 
 
-Feature: Move-Stable Row IDs
+Feature: Stable Row IDs
 ----------------------------
 
-The row ids features assigns a unique u64 id to each row in the table. This id is
-stable after being moved (such as during compaction), but is not necessarily
-stable after a row is updated. (A future feature may make them stable after
-updates.) To make access fast, a secondary index is created that maps row ids to
-their locations in the table. The respective parts of these indices are stored
-in the respective fragment's metadata.
+The row IDs features assigns a unique u64 ID to each row in the table. This ID is
+stable throughout the lifetime of the row. To make access fast, a secondary index
+is created that maps row IDs to their locations in the table. The respective parts
+of these indices are stored in the respective fragment's metadata.
 
-row id
-  A unique auto-incrementing u64 id assigned to each row in the table.
+row ID
+  A unique auto-incrementing u64 ID assigned to each row in the table.
 
 row address
   The current location of a row in the table. This is a u64 that can be thought
-  of as a pair of two u32 values: the fragment id and the local row offset. For
+  of as a pair of two u32 values: the fragment ID and the local row offset. For
   example, if the row address is (42, 9), then the row is in the 42rd fragment
   and is the 10th row in that fragment.
 
-row id sequence
-  The sequence of row ids in a fragment.
+row ID sequence
+  The sequence of row IDs in a fragment.
 
-row id index
-  A secondary index that maps row ids to row addresses. This index is constructed
-  by reading all the row id sequences.
+row ID index
+  A secondary index that maps row IDs to row addresses. This index is constructed
+  by reading all the row ID sequences.
 
-Assigning row ids
+Assigning row IDs
 ~~~~~~~~~~~~~~~~~
 
-Row ids are assigned in a monotonically increasing sequence. The next row id is
+Row IDs are assigned in a monotonically increasing sequence. The next row ID is
 stored in the manifest as the field ``next_row_id``. This starts at zero. When
 making a commit, the writer uses that field to assign row ids to new fragments.
 If the commit fails, the writer will re-read the new ``next_row_id``, update
-the new row ids, and then try again. This is similar to how the ``max_fragment_id``
-is used to assign new fragment ids.
+the new row IDs, and then try again. This is similar to how the ``max_fragment_id``
+is used to assign new fragment IDs.
 
-When a row id updated, it it typically assigned a new row id rather than
-reusing the old one. This is because this feature doesn't have a mechanism to
-update secondary indices that may reference the old values for the row id. By
-deleting the old row id and creating a new one, the secondary indices will avoid
-referencing stale data.
+Moving rows
+~~~~~~~~~~~
+
+When a row is moved to a new file, its row ID is not changed. This happens either
+when the row is updated or just relocated as-is.
+
+When moving rows to new fragments, care must be taken to keep the fragment bitmaps
+up-to-date. First, indexed and unindexed rows should not be mixed in the same
+fragment. So for example, a merge-insert operation should keep new rows in a
+separate fragment from updated rows that are covered by an index. Second, the
+fragment bitmap should be updated to reflect the new location of the row ids.
+Finally, if the indexed value in a row is updated, the row should be considered
+unindexed again.
 
 Row ID sequences
 ~~~~~~~~~~~~~~~~
 
-The row id values for a fragment are stored in a ``RowIdSequence`` protobuf
-message. This is described in the `protos/rowids.proto`_ file. Row id sequences
+The row ID values for a fragment are stored in a ``RowIdSequence`` protobuf
+message. This is described in the `protos/rowids.proto`_ file. Row ID sequences
 are just arrays of u64 values, which have representations optimized for the
 common case where they are sorted and possibly contiguous. For example, a new
-fragment will have a row id sequence that is just a simple range, so it is 
+fragment will have a row ID sequence that is just a simple range, so it is 
 stored as a ``start`` and ``end`` value.
 
 These sequence messages are either stored inline in the fragment metadata, or
@@ -552,13 +558,43 @@ operations.
 Row ID index
 ~~~~~~~~~~~~
 
-To ensure fast access to rows by their row id, a secondary index is created that
-maps row ids to their locations in the table. This index is built when a table is
-loaded, based on the row id sequences in the fragments. For example, if fragment
-42 has a row id sequence of ``[0, 63, 10]``, then the index will have entries for
+To ensure fast access to rows by their row ID, a secondary index is created that
+maps row IDs to their locations in the table. This index is built when a table is
+loaded, based on the row ID sequences in the fragments. For example, if fragment
+42 has a row ID sequence of ``[0, 63, 10]``, then the index will have entries for
 ``0 -> (42, 0)``, ``63 -> (42, 1)``, ``10 -> (42, 2)``. The exact form of this
 index is left up to the implementation, but it should be optimized for fast lookups.
 
 .. _protos/table.proto: https://github.com/lancedb/lance/blob/main/protos/table.proto
 .. _protos/rowids.proto: https://github.com/lancedb/lance/blob/main/protos/rowids.proto
 
+Row ID masks
+~~~~~~~~~~~~
+
+Because index files are immutable, they main contain references to row IDs that
+have been deleted or that have new values. To handle this, a mask is created for
+the index.
+
+.. image:: _static/stable_row_id_indices.png
+
+For example, consider the sequence shown in the above image. It has a dataset
+with two columns, ``str`` and ``vec``. A string column and a vector column. Each
+of them have indices, a scalar index for the string column and a vector index for
+the vector column. There is just one fragment in the dataset, with contiguous row
+IDs 1 through 3.
+
+When an update operation is made that modifes the ``vec`` column in row 2, a
+new fragment is created with the updated value. A deletion file is added to the
+original fragment marking that row 2 as deleted in the first file. In the ``str``
+index, the fragment bitmap is updated to reflect the new location of the row IDs:
+``{1, 2}``. Meanwhile, the ``vec`` index's fragment bitmap does not update, staying
+at ``{1}``. This is because the value in ``vec`` was updated, so the data in the
+index no longer reflects the data in the table.
+
+The combination of the fragment bitmaps and the row ID sequences are used to
+create a mask for each index. To create the mask for the ``vec`` index, the
+row IDs ``{1, 2, 3}`` from fragment 1 are masked by that fragment's deletion file
+to get ``{1, 3}``. That is the only fragment covered by that index, so that is
+the final mask. For the ``str`` index, this operation of subtracting the
+fragment deletion file from the row ID sequence is done for each fragment, and
+then the results are unioned to get the final mask: ``{1, 2, 3}``.

docs/read_and_write.rst

@@ -652,7 +652,7 @@ When files are rewritten, the original row addresses are invalidated. This means
 affected files are no longer part of any ANN index if they were before. Because
 of this, it's recommended to rewrite files before re-building indices.
 
-.. TODO: remove this last comment once move-stable row ids are default.
+.. TODO: remove this last comment once stable row ids are default.
 
 Object Store Configuration
 --------------------------

protos/table.proto

@@ -83,8 +83,7 @@ message Manifest {
   //
   // Known flags:
   // * 1: deletion files are present
-  // * 2: move_stable_row_ids: row IDs are tracked and stable after move operations
-  //       (such as compaction), but not updates.
+  // * 2: row ids are stable and stored as part of the fragment metadata.
   uint64 reader_feature_flags = 9;
 
   // Feature flags for writers.

rust/lance-table/src/format/manifest.rs

@@ -264,8 +264,8 @@ impl Manifest {
         fragments
     }
 
-    /// Whether the dataset uses move-stable row ids.
-    pub fn uses_move_stable_row_ids(&self) -> bool {
+    /// Whether the dataset uses stable row ids.
+    pub fn uses_stable_row_ids(&self) -> bool {
         self.reader_feature_flags & FLAG_MOVE_STABLE_ROW_IDS != 0
     }
 

rust/lance/src/dataset.rs

@@ -569,7 +569,7 @@ impl Dataset {
         );
 
         let manifest_config = ManifestWriteConfig {
-            use_move_stable_row_ids: params.enable_move_stable_row_ids,
+            use_move_stable_row_ids: params.enable_stable_row_ids,
             storage_format: Some(DataStorageFormat::new(storage_version)),
             ..Default::default()
         };
@@ -2538,7 +2538,7 @@ mod tests {
             test_uri,
             Some(WriteParams {
                 data_storage_version: Some(data_storage_version),
-                enable_move_stable_row_ids: use_stable_row_id,
+                enable_stable_row_ids: use_stable_row_id,
                 ..Default::default()
             }),
         )
@@ -2649,7 +2649,7 @@ mod tests {
         let write_params = WriteParams {
             mode: WriteMode::Append,
             data_storage_version: Some(data_storage_version),
-            enable_move_stable_row_ids: use_stable_row_id,
+            enable_stable_row_ids: use_stable_row_id,
             ..Default::default()
         };
 
@@ -2762,7 +2762,7 @@ mod tests {
             data_storage_version: Some(data_storage_version),
             max_rows_per_file: 1024,
             max_rows_per_group: 150,
-            enable_move_stable_row_ids: use_stable_row_id,
+            enable_stable_row_ids: use_stable_row_id,
             ..Default::default()
         };
         Dataset::write(data, test_uri, Some(write_params.clone()))
@@ -3204,7 +3204,7 @@ mod tests {
             test_uri,
             Some(WriteParams {
                 data_storage_version: Some(data_storage_version),
-                enable_move_stable_row_ids: use_stable_row_id,
+                enable_stable_row_ids: use_stable_row_id,
                 ..Default::default()
             }),
         )

rust/lance/src/dataset/fragment.rs

@@ -480,7 +480,7 @@ impl FileFragment {
         let deletion_vec_load =
             self.load_deletion_vector(&self.dataset.object_store, &self.metadata);
 
-        let row_id_load = if self.dataset.manifest.uses_move_stable_row_ids() {
+        let row_id_load = if self.dataset.manifest.uses_stable_row_ids() {
             futures::future::Either::Left(
                 load_row_id_sequence(&self.dataset, &self.metadata).map_ok(Some),
             )
@@ -1273,7 +1273,7 @@ pub struct FragmentReader {
 
     /// The row id sequence
     ///
-    /// Only populated if the move-stable row id feature is enabled.
+    /// Only populated if the stable row id feature is enabled.
     row_id_sequence: Option<Arc<RowIdSequence>>,
 
     /// ID of the fragment

rust/lance/src/dataset/optimize.rs

@@ -627,8 +627,8 @@ async fn rewrite_files(
     // num deletions recorded. If that's the case, we need to grab and set that
     // information.
     let fragments = migrate_fragments(dataset.as_ref(), &task.fragments, recompute_stats).await?;
-    // If we aren't using move-stable row ids, then we need to remap indices.
-    let needs_remapping = !dataset.manifest.uses_move_stable_row_ids();
+    // If we aren't using stable row ids, then we need to remap indices.
+    let needs_remapping = !dataset.manifest.uses_stable_row_ids();
     let mut scanner = dataset.scan();
     scanner
         .with_fragments(fragments.clone())
@@ -759,8 +759,8 @@ pub async fn commit_compaction(
         return Ok(CompactionMetrics::default());
     }
 
-    // If we aren't using move-stable row ids, then we need to remap indices.
-    let needs_remapping = !dataset.manifest.uses_move_stable_row_ids();
+    // If we aren't using stable row ids, then we need to remap indices.
+    let needs_remapping = !dataset.manifest.uses_stable_row_ids();
 
     let mut rewrite_groups = Vec::with_capacity(completed_tasks.len());
     let mut metrics = CompactionMetrics::default();
@@ -1450,7 +1450,7 @@ mod tests {
         let write_params = WriteParams {
             max_rows_per_file: 1000,
             data_storage_version: Some(data_storage_version),
-            enable_move_stable_row_ids: use_stable_row_id,
+            enable_stable_row_ids: use_stable_row_id,
             ..Default::default()
         };
         let mut dataset = Dataset::write(reader, test_uri, Some(write_params))
@@ -1518,14 +1518,14 @@ mod tests {
         }
 
         assert_eq!(
-            dataset.manifest.uses_move_stable_row_ids(),
+            dataset.manifest.uses_stable_row_ids(),
             use_stable_row_id,
         );
     }
 
     #[tokio::test]
     async fn test_stable_row_indices() {
-        // Validate behavior of indices after compaction with move-stable row ids.
+        // Validate behavior of indices after compaction with stable row ids.
         let mut data_gen = BatchGenerator::new()
             .col(Box::new(
                 RandomVector::new().vec_width(128).named("vec".to_owned()),
@@ -1535,7 +1535,7 @@ mod tests {
             data_gen.batch(5_000),
             "memory://test/table",
             Some(WriteParams {
-                enable_move_stable_row_ids: true,
+                enable_stable_row_ids: true,
                 max_rows_per_file: 1_000, // 5 files
                 ..Default::default()
             }),
@@ -1613,7 +1613,7 @@ mod tests {
         let _metrics = compact_files(&mut dataset, options, None).await.unwrap();
 
         // The indices should be unchanged after compaction, since we are using
-        // move-stable row ids.
+        // stable row ids.
         let current_indices = index_set(&dataset).await;
         assert_eq!(indices, current_indices);
 

rust/lance/src/dataset/optimize/remapping.rs

@@ -1,7 +1,7 @@
 // SPDX-License-Identifier: Apache-2.0
 // SPDX-FileCopyrightText: Copyright The Lance Authors
 
-//! Utilities for remapping row ids. Necessary before move-stable row ids.
+//! Utilities for remapping row ids. Necessary before stable row ids.
 //!
 
 use async_trait::async_trait;