Make payment_key derivation deterministic

[tnull]

Previously, KeysManager::derive_channel_keys would derive the channel's payment_key uniquely on a per-channel basis which would disallow users losing their channel_keys_id to recover funds. As there is no necessity to have payment_key derivation depend on channel_keys_id we can allow for easier recovery of any non-HTLC encumbered funds if we make payment_key derivation deterministic.

Here we do just this but also include a larger refactor introducing a ChannelKeysDerivationParameters struct holding a versioning field to be able to express this change in the derivation scheme. To this end we ensure that legacy channels will continue to use the old derivation scheme after upgrade, while new channels will derive the payment_key
deterministicly.

To this end, we use the first byte of channel_keys_id as a versioning byte indicating the version of the used channel keys derivation scheme. Note that Previously KeysManager::generate_channel_keys_id would with
very high likelyhood never have generated a channel_keys_id with a non-null first byte, which makes this a backwards-compatible change for any users that didn't run custom implementations of
SignerProvider::generate_channel_keys_id conflicting with this assumption.

Apart from this, we add some refactoring commits that drop unused logic (Writeable for InMemorySigner) and rename fields to align them with the BOLTs for clarity.

We previously also discussed abusing the first byte of channel_keys_id as a version byte, which wouldn't require the overhead of introducing ChannelKeysDerivationParameters and serializing the channel_keys_derivation_version field independently everywhere. However, as discussed I now chose to go with the latter approach as it seems cleaner and more future-proof.

That said, looking for an approach ACK here before continuing. Will be draft until then.

Should be good for review

TODO:

• Fix one remaining failing test case.
• Add test case for (partial) recovery from seed only.
• Include commit adding a channel_keys_id argument to get_shutdown_scriptpubkey, just as for the destination case. (cc @alecchendev)

[tnull]

After discussion this offline I now reverted this to simply use the first byte of channel_keys_id as a versioning field which considerably reduces the diff size. Note however that this approach might have backwards compatibility issues if users would have custom implementations of SignerProvider::generate_channel_keys_id for which the always-0-first-byte assumption doesn't hold.

Furthermore, this change will of course prohibit downgrading channel monitors to use the older derivation version. This is also the reason why currently monitor_tests::test_restored_packages_retry was failing before. Now added a workaround to use the legacy keys derivation for this test.

[codecov]

Codecov Report

Attention: Patch coverage is 94.95798% with 6 lines in your changes missing coverage. Please review.

Project coverage is 89.72%. Comparing base (8c086c7) to head (8991d74).
Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/sign/mod.rs	94.82%	2 Missing and 1 partial ⚠️
lightning/src/ln/channel.rs	81.81%	2 Missing ⚠️
lightning/src/util/test_utils.rs	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3391      +/-   ##
==========================================
+ Coverage   89.66%   89.72%   +0.05%     
==========================================
  Files         128      128              
  Lines      104887   105577     +690     
  Branches   104887   105577     +690     
==========================================
+ Hits        94052    94734     +682     
- Misses       8126     8140      +14     
+ Partials     2709     2703       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.