Add MigratableKVStore trait

[tnull]

Closes #3387

We add a MigratableKVStore trait (suggestion for better naming are welcome) that describes methods required to write generalized KVStore-to-KVStore migration logic. In particular, it adds afn list_all_keys method that exhaustively lists all keys stored, which then can be used by a utility to transfer all of them to the new store before deleting the old state.

We implement this new trait for FilesystemStore.

TODOs (draft until then):

• Add simple migration utility function.
• Add test code for FilesystemStore to FilesystemStore migration.

Should be ready for review.

[tnull]

Added a migration util method and a test for FilesystemStore-to-FilesystemStore migration. Should be ready for review now.

[tnull]

Addressed pending comments, let me know if I can squash fixups.

[G8XSU]

Lgtm, mod nits.

[tnull]

Addressed pending feedback.

[TheBlueMatt]

Feel free to squash, IMO.

[tnull]

Feel free to squash, IMO.

Squashed fixups without further changes.

[codecov]

Codecov Report

Attention: Patch coverage is 54.48718% with 71 lines in your changes missing coverage. Please review.

Project coverage is 91.28%. Comparing base (fe1cf69) to head (3804d4c).
Report is 51 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning-persister/src/fs_store.rs	41.88%	51 Missing and 17 partials ⚠️
lightning/src/util/persist.rs	66.66%	0 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3481      +/-   ##
==========================================
+ Coverage   89.70%   91.28%   +1.58%     
==========================================
  Files         130      130              
  Lines      107422   118991   +11569     
  Branches   107422   118991   +11569     
==========================================
+ Hits        96362   108622   +12260     
+ Misses       8669     8136     -533     
+ Partials     2391     2233     -158     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[G8XSU]

Lgtm!

[tnull]

Force-pushed with fixups:

2024-12-add-kvstore-migration-ext ~/workspace/rust-lightning> git diff-tree -U2 3804d4c3d b0af39faa
diff --git a/lightning-persister/src/fs_store.rs b/lightning-persister/src/fs_store.rs
index a1d13c06b..487e1a7ac 100644
--- a/lightning-persister/src/fs_store.rs
+++ b/lightning-persister/src/fs_store.rs
@@ -348,6 +348,9 @@ fn dir_entry_is_key(p: &Path) -> Result<bool, lightning::io::Error> {

        let metadata = p.metadata().map_err(|e| {
-               let msg =
-                       format!("Failed to list keys at path {:?}: {}", p.to_str().map(PrintableString), e);
+               let msg = format!(
+                       "Failed to list keys at path {}: {}",
+                       PrintableString(p.to_str().unwrap_or_default()),
+                       e
+               );
                lightning::io::Error::new(lightning::io::ErrorKind::Other, msg)
        })?;
@@ -362,10 +365,10 @@ fn dir_entry_is_key(p: &Path) -> Result<bool, lightning::io::Error> {
                debug_assert!(
                        false,
-                       "Failed to list keys at path {:?}: file couldn't be accessed.",
-                       p.to_str().map(PrintableString)
+                       "Failed to list keys at path {}: file couldn't be accessed.",
+                       PrintableString(p.to_str().unwrap_or_default())
                );
                let msg = format!(
-                       "Failed to list keys at path {:?}: file couldn't be accessed.",
-                       p.to_str().map(PrintableString)
+                       "Failed to list keys at path {}: file couldn't be accessed.",
+                       PrintableString(p.to_str().unwrap_or_default())
                );
                return Err(lightning::io::Error::new(lightning::io::ErrorKind::Other, msg));
@@ -384,10 +387,10 @@ fn get_key_from_dir_entry(p: &Path, base_path: &Path) -> Result<String, lightnin
                                        debug_assert!(
                                                false,
-                                               "Failed to list keys of path {:?}: file path is not valid key",
-                                               p.to_str().map(PrintableString)
+                                               "Failed to list keys of path {}: file path is not valid key",
+                                               PrintableString(p.to_str().unwrap_or_default())
                                        );
                                        let msg = format!(
-                                               "Failed to list keys of path {:?}: file path is not valid key",
-                                               p.to_str().map(PrintableString)
+                                               "Failed to list keys of path {}: file path is not valid key",
+                                               PrintableString(p.to_str().unwrap_or_default())
                                        );
                                        return Err(lightning::io::Error::new(lightning::io::ErrorKind::Other, msg));
@@ -396,10 +399,10 @@ fn get_key_from_dir_entry(p: &Path, base_path: &Path) -> Result<String, lightnin
                                debug_assert!(
                                        false,
-                                       "Failed to list keys of path {:?}: file path is not valid UTF-8",
-                                       p
+                                       "Failed to list keys of path {}: file path is not valid UTF-8",
+                                       PrintableString(p.to_str().unwrap_or_default())
                                );
                                let msg = format!(
-                                       "Failed to list keys of path {:?}: file path is not valid UTF-8",
-                                       p.to_str().map(PrintableString)
+                                       "Failed to list keys of path {}: file path is not valid UTF-8",
+                                       PrintableString(p.to_str().unwrap_or_default())
                                );
                                return Err(lightning::io::Error::new(lightning::io::ErrorKind::Other, msg));
@@ -409,10 +412,13 @@ fn get_key_from_dir_entry(p: &Path, base_path: &Path) -> Result<String, lightnin
                        debug_assert!(
                                false,
-                               "Failed to list keys of path {:?}: {}",
-                               p.to_str().map(PrintableString),
+                               "Failed to list keys of path {}: {}",
+                               PrintableString(p.to_str().unwrap_or_default()),
+                               e
+                       );
+                       let msg = format!(
+                               "Failed to list keys of path {}: {}",
+                               PrintableString(p.to_str().unwrap_or_default()),
                                e
                        );
-                       let msg =
-                               format!("Failed to list keys of path {:?}: {}", p.to_str().map(PrintableString), e);
                        return Err(lightning::io::Error::new(lightning::io::ErrorKind::Other, msg));
                },
@@ -467,10 +473,10 @@ impl MigratableKVStore for FilesystemStore {
                                                debug_assert!(
                                                        false,
-                                                       "Failed to list keys of path {:?}: only two levels of namespaces are supported",
-                                                       tertiary_path.to_str()
-                                                       );
+                                                       "Failed to list keys of path {}: only two levels of namespaces are supported",
+                                                       PrintableString(tertiary_path.to_str().unwrap_or_default())
+                                               );
                                                let msg = format!(
-                                                       "Failed to list keys of path {:?}: only two levels of namespaces are supported",
-                                                       tertiary_path.to_str()
+                                                       "Failed to list keys of path {}: only two levels of namespaces are supported",
+                                                       PrintableString(tertiary_path.to_str().unwrap_or_default())
                                                );
                                                return Err(lightning::io::Error::new(

[TheBlueMatt]

Can land this and fix up the test later, or feel free to fix it now.