lightning-liquidity persistence: Add serialization logic for services and event queue
#4059
Conversation
|
👋 Thanks for assigning @TheBlueMatt as a reviewer! |
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
2 times, most recently
from
124211d to
26f3ce3
Compare
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
4 times, most recently
from
a98dff6 to
d630c4e
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4059 +/- ##
==========================================
- Coverage 88.33% 88.09% -0.24%
==========================================
Files 177 179 +2
Lines 131896 132736 +840
Branches 131896 132736 +840
==========================================
+ Hits 116512 116938 +426
- Misses 12728 13116 +388
- Partials 2656 2682 +26
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
d630c4e to
70118e7
Compare
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
70118e7 to
dd43edc
Compare
|
this all LGTM. I have a small concern: maybe I’m being a little paranoid, but read_lsps2_service_peer_states and read_lsps5_service_peer_states pull every entry from the KVStore into memory with no limit. That could lead to unbounded state, exhausting memory and crash. Maybe we can add a limit on how many entries we load into memory to protect against this dos? not sure how realistic this is though. maybe an attacker could have access to or share the same storage with the victim, and they could dump effectively infinite data onto disk. in this scenario, probably the victim would be vulnerable to other attacks too, but still.. |
Reading state from disk (currently) happens on startup only, so crashing wouldn't be the worst thing, we would simply fail to start up properly. Some even argue that we need to panic if we hit any IO errors at this point to escalate to an operator. We could add some safeguard/upper bound, but I'm honestly not sure what it would protect against.
Heh, well, if we assume the attacker has write access to our |
|
🔔 1st Reminder Hey @TheBlueMatt! This PR has been waiting for your review. |
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
dd43edc to
f73146b
Compare
| @@ -45,6 +46,10 @@ pub struct LSPS2GetInfoRequest { | |||
| pub token: Option<String>, | |||
| } | |||
|
|
|||
| impl_writeable_tlv_based!(LSPS2GetInfoRequest, { | |||
There was a problem hiding this comment.
Do we really want to have two ways to serialize all these types? Wouldn't it make more sense to just use the serde serialization we already have and wrap that so that it can't all be misused?
There was a problem hiding this comment.
Yes, I think I'd be in favor of using TLV serialization for our own persistence.
Note that the compat guarantees of LSPS0/the JSON/serde format might not exactly match what we require in LDK, and our Rust representation might also diverge from the pure JSON impl. On top of that JSON is of course much less efficient.
There was a problem hiding this comment.
Hmm, is there some easy way to avoid exposing that in the public API, then? Maybe a wrapper struct oe extension trait for serialization somehow? Seems like kinda a footgun for users, I think?
There was a problem hiding this comment.
Hmm, is there some easy way to avoid exposing that in the public API, then? Maybe a wrapper struct oe extension trait for serialization somehow? Seems like kinda a footgun for users, I think?
Not quite sure I understand the footgun? You mean because these types then have Writeable as well as Serialize implementations on them and users might wrongly pick Writeable when they use the types independently from/outside of lightning-liquidity?
There was a problem hiding this comment.
Sure, for example. Someone who uses serde presumably has some wrapper that serde-writes Writeable structs and suddenly their code could read/compile totally fine and be reading the wrong kind of thing. If they have some less-used codepaths (eg writing Events before they process them and then removing them again after) they might not find immediately.
There was a problem hiding this comment.
Sure, for example. Someone who uses serde presumably has some wrapper that serde-writes
Writeablestructs and suddenly their code could read/compile totally fine and be reading the wrong kind of thing.
I'm confused - Writeable is an LDK concept not connected to serde? Do you mean Serialize? But that also has completely separate API? So how would they trip up? You mean they'd confuse Writeable and Serialize?
| ) -> Pin<Box<dyn Future<Output = Result<(), lightning::io::Error>> + Send>> { | ||
| let outer_state_lock = self.per_peer_state.read().unwrap(); | ||
| let mut futures = Vec::new(); | ||
| for (counterparty_node_id, peer_state) in outer_state_lock.iter() { |
There was a problem hiding this comment.
Huh? Why would we ever want to do a single huge persist pass and write every peer's state at once? Shouldn't we be doing this iteratively? Same applies in the LSPS2 service.
There was a problem hiding this comment.
Yes, only persisting what's needed/changed will be part of the next PR as it ties into how we wake the BP to drive persistence (cf. "Avoid re-persisting peer states if no changes happened (needs_persist flag everywhere)" bullet over at #4058 (comment)).
There was a problem hiding this comment.
I'm confused why we're adding this method then? If its going to be removed in the next PR in the series we should just not add it in the first place.
There was a problem hiding this comment.
No, it's not gonna be removed, but extended: PeerState (here as well as in LSPS2) will gain a dirty/needs_persist flag and we'd simply skip persisting any entries that haven't been changed since the last persistence round.
There was a problem hiding this comment.
That seems like a weird design if we need to persist something immediately while its being operated on - we have the node in question why walk a whole peer list? Can you put up the followup code so we can see how its going to be used? Given this PR is mostly boilerplate I honestly wouldn't mind it being a bit bigger, as long as the code isn't too crazy.
There was a problem hiding this comment.
That seems like a weird design if we need to persist something immediately while its being operated on - we have the node in question why walk a whole peer list?
Yes, this is why persist_peer_state is a separate method - for inline persistence where we already hold the lock to the peer state we'd just call that. For the general/eventual persistence the background processor task calls LiquidityManager::persist which calls through to the respective LSPS*ServiceHandler::persist methods which then only persists the entries marked dirty since the last persistence round.
Can you put up the followup code so we can see how its going to be used? Given this PR is mostly boilerplate I honestly wouldn't mind it being a bit bigger, as long as the code isn't too crazy.
Sure will do as soon as it's ready an in a coherent state, although I had hoped to land this PR this week.
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
f73146b to
2971982
Compare
|
Rebased to address minor conflict. |
|
🔔 1st Reminder Hey @TheBlueMatt! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @TheBlueMatt! This PR has been waiting for your review. |
|
Excuse the delay here, got a bit side-tracked with other things end of last week. Given that the feedback was in favor of a single mega-PR, I'm now working to finish everything up and adding the remaining tasks of #4058 to this PR. |
We add `KVStore` to `LiquidityManager`, which will be used in the next commits. We also add a `LiquidityManagerSync` wrapper that wraps a the `LiquidityManager` interface which will soon become async due to usage of the async `KVStore`.
.. this is likely only temporary necessary as we can drop our own `dummy_waker` implementation once we bump MSRV.
We add simple `persist` call to `LSPS2ServiceHandler` that sequentially persist all the peer states under a key that encodes their node id.
We add simple `persist` call to `LSPS5ServiceHandler` that sequentially persist all the peer states under a key that encodes their node id.
We add simple `persist` call to `EventQueue` that persists it under a `event_queue` key.
We read any previously-persisted state upon construction of `LiquidityManager`.
We read any previously-persisted state upon construction of `LiquidityManager`.
We read any previously-persisted state upon construction of `LiquidityManager`.
We let the background processor task regularly call `LiquidityManger::persist`. We also change the semantics of the `Future` for waking the background processor to also be used when we need repersisting (which we'll do in the next commit).
.. we only persist the event queue if necessary and wake the BP to do so when something changes.
.. we only persist the service handler if necessary.
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
0367a27 to
849c857
Compare
|
Alright, now updated to:
The ~only thing left is end-to-end test coverage for persistence which I intend to add (here or in a follow-up) in the coming days. I also opened two new issues for larger refactorings we should do some time after this lands (#4073/#4074). |
.. to allow access in a non-async context
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
3 times, most recently
from
d61c297 to
9c3ea4f
Compare
.. and wrap them accordingly for the `LSPS2ServiceHandlerSync` variant.
.. we only persist the service handler if necessary.
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
9c3ea4f to
9cdf6c9
Compare
This is the second PR in a series of PRs adding persistence tolightning-liquidity(see #4058). As this is already >1000LoC, I now decided to put this up as an intermediary step instead of adding everything in one go.In this PR we add the serialization logic for for the LSPS2 and LSPS5 service handlers as well as for the event queue. We also have
LiquidityManagertake aKVStoretowards which it persists the respective peer states keyed by the counterparty's node id.LiquidityManager::newnow also deserializes any previously-persisted state from that givenKVStore.We then have
BackgroundProcessordrive persistence, skip persistence for unchanged LSPS2/LSPS5PeerStates, and useasyncinline persistence forLSPS2ServiceHandlerwhere needed.This also adds a bunch of boilerplate to account for both
KVStoreandKVStoreSyncvariants, following the approach we previously took withOutputSweeperetc.cc @martinsaposnic