change(state): Check database format is valid every 5 minutes, to cat…

…ch format errors in new block code (#7602)

* Refactor format checks, stop running quick check before upgrade

* Speed up startup by always running the format checks in parallel

* Make detailed format checks cancellable on shutdown

* Prepare for periodic format checks

* Time upgrades and validity checks

* Run a database format check every 5 minutes

* Wait for a cancel signal rather than unconditionally sleeping

* Move check_max_on_disk_height() into the format checks

* Move spawn_format_change() into its own method

zebra-state/src/constants.rs

@@ -1,5 +1,7 @@
 //! Constants that impact state behaviour.
 
+use std::time::Duration;
+
 use lazy_static::lazy_static;
 use regex::Regex;
 use semver::Version;
@@ -67,6 +69,12 @@ pub fn latest_version_for_adding_subtrees() -> Version {
 /// Use [`Config::version_file_path()`] to get the path to this file.
 pub(crate) const DATABASE_FORMAT_VERSION_FILE_NAME: &str = "version";
 
+/// The interval between database format checks for newly added blocks.
+///
+/// This should be short enough that format bugs cause CI test failures,
+/// but long enough that it doesn't impact performance.
+pub(crate) const DATABASE_FORMAT_CHECK_INTERVAL: Duration = Duration::from_secs(5 * 60);
+
 /// The maximum number of blocks to check for NU5 transactions,
 /// before we assume we are on a pre-NU5 legacy chain.
 ///