neutrino remove sweeptx #7800

ziggie1984 · 2023-07-01T10:30:59Z

Change Description

Up until now we were only removing pending sweep inputs when a transaction was confirmed spending an input of the same exclusive group. For neutrino backends this is not enough, we need to make sure that the whole unconfirmed transaction is purged as well. This refers especially to anchor force closes where up to 3 sweep inputs are broadcasted within the same exclusive group. In the worst case scenario 2 additional inputs will be blocked because they are used to sweep anchor outputs which will never confirm because their parent will never make it into the blockchain as soon as one of them is confirmed.
This change does not restrict the removing soley for neutrino backends because it is most likely a NOOP for other backends. Nonetheless I think it does no harm and prevents us from surfacing the backend-structure into the Sweeper package. Open for suggestions here if there is a better way.

Restarting the node will not fix this behaviour because the transaction is still in the unminded bucket which means it will be rebroadcasted and bitcoind nodes will not fail this transaction because they will consider it as orphan.

Enhanced an itest to easily verify this:

make itest icase=multi_hop_local_force_close_on-chain_htlc_timeout/zeroconf=false/committype=ANCHORS backend=neutrino

Steps to Test

Steps for reviewers to follow to test the change.

Pull Request Checklist

Testing

Your PR passes all CI checks.
Tests covering the positive and negative (error paths) are included.
Bug fixes contain tests triggering the bug to prevent regressions.

Code Style and Documentation

The change obeys the Code Documentation and Commenting guidelines, and lines wrap at 80.
Commits follow the Ideal Git Commit Structure.
Any new logging statements use an appropriate subsystem and logging level.
There is a change description in the release notes, or [skip ci] in the commit message for small changes.

📝 Please see our Contribution Guidelines for further guidance.

ziggie1984 · 2023-07-01T11:04:05Z

A switch from neutrino to normal backends could solve the problem too, so maybe that's the reason why it comes up so late?

While analysing the bug I realised that we also do not save the "ForceClosing" transactions (lncli listchaintxns) when they are confirmed. Maybe the reason for it is that all of the outputs are not part of the Default Wallet ? Do we want fix this too?

guggero

Thanks a lot for catching this and providing a fix!
I think this makes sense, but would like to get at least another review from someone who is a bit more familiar with the internals of the sweeper.

sweep/sweeper.go

lntest/harness_assertion.go

sweep/sweeper.go

Roasbeef · 2023-07-11T17:46:00Z

Isn't this equivalent to just removing the !isOurTx in this logic block: https://github.com/lightningnetwork/lnd/blob/9ff2d4005f9ec24d3fe56f6e762a6c2a082e2a01/sweep/sweeper.go#L706-L728

ziggie1984 · 2023-07-11T18:40:21Z

Isn't this equivalent to just removing the !isOurTx in this logic block:

Good idea but our use case is a bit different here. So when we launch our CPFP anchor sweeps we do that in the worst case for 3 different commitment txids, (local,remote,remote_dangling) in your function only the inputs are checked which are spent in the transaction, however we must trigger the removal also when sweeps of the same exclusive group are confirmed, because only one commitment tx can make it into the blockchain.

Maybe there is a way to include this removal in this code, instead of introducing another function.

Ok so the interesting part happens here, where all pendingsweeps with the same exclusive group are evaluated and exactly there we have to hook in for the removal of those lingering unminded transaction:
https://github.com/lightningnetwork/lnd/blob/9ff2d4005f9ec24d3fe56f6e762a6c2a082e2a01/sweep/sweeper.go#L756C1-L763.

So I think it's the safest way to include the call in the s.removeExclusiveGroup so that we can be sure the relevant tx is removed?

ziggie1984 · 2023-08-09T18:42:44Z

Regarding our solution in #7879 we should also make sure that we remove invalid transactions when our sweeper exhausts for specific outputs. This would make sure in a case of the anchorsweeps or any other sweep also gets removed from the unminded bucket of the wallet and the Rebroadcaster as well. Basically calling the new RemoveConflict function when we signalAndRemove from the sweeper store.

The removal of transactions in the same exclusive group could also be fixed in another way. So we could add an ancestor field to the ChainNotifier and I think we would also need to add it into btcwallet which also needs to be aware in case an ancestor (in our case the funding output) is mined. Would be a bigger change but maybe more elegant because we actually move things more closely to the software stack where they belong? Wdyt ?

ziggie1984 · 2023-08-09T18:53:34Z

The question here which is still open: How do we remove transactions resulting from this behaviour prior to this fix? Where the channel is already resolved and no more sweeps are registered but the transaction is still stuck in the UnmindedTx bucket ? Offering an Endpoint where noderunner could manually purge UnminedTransactions ? Open for suggestions

ziggie1984 · 2023-08-10T12:39:35Z

The question here which is still open: How do we remove transactions resulting from this behaviour prior to this fix? Where the channel is already resolved and no more sweeps are registered but the transaction is still stuck in the UnmindedTx bucket ? Offering an Endpoint where noderunner could manually purge UnminedTransactions ? Open for suggestions

Rescanning the wallet works here 🙌 user will need to trigger a rescan. However we can maybe add an rpc endpoint nonetheless, facilitating the process in case the user knows what he is missing ?

Roasbeef · 2023-10-03T17:44:34Z

However we can maybe add an rpc endpoint nonetheless, facilitating the process in case the user knows what he is missing ?

@ziggie1984 I think that's a good idea. Having to add that flag, then also resume with a special unlock flag is kinda annoying. Should be a small change.

lightninglabs-deploy · 2023-10-31T18:48:18Z

@yyforyongyu: review reminder
@ziggie1984, remember to re-request review from reviewers when ready

ziggie1984 · 2023-10-31T20:10:40Z

!lightninglabs-deploy mute

ziggie1984 · 2023-11-07T12:08:49Z

Added a new walletrpc endpoint RemoveTransaction to easily remove an unconfirmed transaction from the default wallet.

Only relevant for neutrino backends imo, prior to that people could just use lncli wallet publishtx which would have removed the transaction if it was not successfully registered with the mempool.

yyforyongyu

Nice addition of the RemoveTransaction! My main question is whether we should combine removeLastSweepDescendants and removeConflictingSweepTx.

sweep/sweeper.go

itest/lnd_onchain_test.go

cmd/lncli/walletrpc_active.go

yyforyongyu

Looking good🏖 Just a few nits. Meanwhile do you think the following case is fixed here?

lnd/itest/lnd_onchain_test.go

Lines 523 to 551 in d9b88fb

    
           // Send another UTXO if this is a neutrino backend. When sweeping 
        
           // anchors, there are two transactions created, `local_sweep_tx` for 
        
           // sweeping Alice's anchor on the local commitment, `remote_sweep_tx` 
        
           // for sweeping her anchor on the remote commitment. Whenever the force 
        
           // close transaction is published, Alice will always create these two 
        
           // transactions to sweep her anchor. 
        
           // On the other hand, when creating the sweep txes, the anchor itself 
        
           // is not able to cover the fee, so another wallet UTXO is needed. In 
        
           // our test case, there's a change output that can be used from the 
        
           // above funding process. And it's used by both sweep txes - when `lnd` 
        
           // happens to create the `remote_sweep_tx` first, it will receive an 
        
           // error since its parent tx, the remote commitment, is not known, 
        
           // hence freeing the change output to be used by `local_sweep_tx`. 
        
           // For neutrino client, however, it will consider the transaction which 
        
           // sweeps the remote anchor as an orphan tx, and it will neither send 
        
           // it to the mempool nor return an error to free the change output. 
        
           // Thus, if the change output is already used in `remote_sweep_tx`, we 
        
           // won't have UTXO to create `local_sweep_tx`. 
        
           // 
        
           // NOTE: the order of the sweep requests for the two anchors cannot be 
        
           // guaranteed. If the sweeper happens to sweep the remote anchor first, 
        
           // then the test won't pass without the extra UTXO, which is the source 
        
           // of the flakeness. 
        
           // 
        
           // TODO(yy): make a RPC server for sweeper so we can explicitly check 
        
           // and control its state. 
        
           if ht.IsNeutrinoBackend() { 
        
           	ht.FundCoins(anchorFeeBuffer, alice) 
        
           }

Was thinking how we can test this new behavior for neutrino backend, and this the above case won't be fixed becase these conflicting transactions are only removed AFTER the sweeping tx is confirmed?

sweep/sweeper.go

docs/release-notes/release-notes-0.18.0.md

itest/lnd_onchain_test.go

ziggie1984 · 2023-11-20T18:29:55Z

Was thinking how we can test this new behavior for neutrino backend, and this the above case won't be fixed becase these conflicting transactions are only removed AFTER the sweeping tx is confirmed?

I agree it's not fixed because this only removes them if the commitment is confirmed. But the flakiness is fixed when adding this additional input correct ?

yyforyongyu

LGTM🙏

But the flakiness is fixed when adding this additional input correct ?

Correct.

docs/code_formatting_rules.md

guggero

Did a first pass. Very nice, definitely very useful to have!
Main ask is to change the RPC/CLI a bit to allow only specifying the TXID. Other than that it's mostly nits.

sweep/sweeper.go

lnrpc/walletrpc/walletkit.proto

itest/lnd_onchain_test.go

lnrpc/walletrpc/walletkit_server.go

cmd/lncli/walletrpc_active.go

docs/code_formatting_rules.md

guggero

Great work, we're almost there.
Just one more ask (simplification because of new wallet method we have) and a couple of nits.

guggero · 2023-12-12T11:11:45Z

lnrpc/walletrpc/walletkit_server.go

+	// `ListTransactionDetails` function so we provide an empty string.
+	// -1 equals the mempool height so only unconfirmed transactions are
+	// returned.
+	transactions, err := w.cfg.Wallet.ListTransactionDetails(-1, -1, "")


We can now fetch a transaction directly from the wallet, see #7654.

I wonder if we also should allow to remove confirmed transactions to shortcircuit the wallet rescan, but I guess for this we would also need to introduce a addTx cmd, so that that we can manually add some utxos if we know for sure that they are ours?

see for example issue #8251, where they kind of know what they are missing ?

Yeah, not sure if that's a good idea... You could really mess up your wallet state with that. At the very least it shouldn't be part of this PR anyway so we can properly discuss the feature (and its consequences) in a standalone PR.

cmd/lncli/walletrpc_active.go

docs/code_formatting_rules.md

For anchor channels and neutrino backends we need to make sure that sweeps of the same exclusive group are removed when one of them is confirmed. Otherwise for neutrino backends those sweep transaction are always rebroadcasted and are blocking funds in the worst case scenario.

The RemoveTransaction endpoint removes the transaction with the provided txid including all its descendants from the internal wallet. We still keep watching for the address of the transation in case the transcation is confirmed nonetheless. This command is particular useful for neutrino backends because new bitcoind versions do not reply with an invalid transaction error code when the tx published fails to be included into the mempool (fullnodes do).

This new command calls the new rpc endpoint RemoveTransaction.

ziggie1984 · 2023-12-12T13:23:16Z

lnrpc/walletrpc/walletkit_server.go

+	// as a transaction is confirmed it will be evaluated by the wallet
+	// again and the wallet state would be updated in case the user had
+	// removed the transaction accidentally.
+	if res.NumConfirmations > 0 {


We might think of a --i_know_what_i_am_doing flag and also allow the removal of confirmed txids ?

See my comment above about doing it in a separate PR (if even...). Also, when it comes to naming, I think we'd want something more along the lines of --danger-allow-confirmed since apparently the "I know what I'm doing" approach doesn't seem to work for some users. Psychologically it seems people are more inclined to think they know what they're doing while if something is labelled as "dangerous" they might be more careful (even if they happen to actually know what they are doing).

well said, I observed the same behavior :)

guggero

LGTM 🎉

ziggie1984 force-pushed the neutrino-remove-sweeptx branch 3 times, most recently from 9f32082 to 0e0c49f Compare July 1, 2023 10:59

saubyk added this to the v0.17.1 milestone Jul 1, 2023

ziggie1984 force-pushed the neutrino-remove-sweeptx branch from 0e0c49f to 7dd93dc Compare July 1, 2023 21:07

ziggie1984 marked this pull request as ready for review July 1, 2023 21:13

guggero reviewed Jul 11, 2023

View reviewed changes

ziggie1984 force-pushed the neutrino-remove-sweeptx branch from 7dd93dc to 9ff2d40 Compare July 11, 2023 12:51

ziggie1984 mentioned this pull request Jul 24, 2023

Last sweep tx is constantly rebroadcasted on startup #7708

Closed

saubyk modified the milestones: High Priority, v0.18.0 Aug 4, 2023

saubyk assigned ziggie1984 Aug 8, 2023

yyforyongyu self-requested a review August 9, 2023 04:32

ziggie1984 force-pushed the neutrino-remove-sweeptx branch from 9ff2d40 to 8b85c0c Compare November 7, 2023 12:07

ziggie1984 force-pushed the neutrino-remove-sweeptx branch 2 times, most recently from 0c80433 to 93ed71b Compare November 7, 2023 12:41

ziggie1984 requested a review from guggero November 7, 2023 12:44

ziggie1984 force-pushed the neutrino-remove-sweeptx branch 2 times, most recently from 713bf28 to def84d2 Compare November 7, 2023 15:18

yyforyongyu requested changes Nov 14, 2023

View reviewed changes

ziggie1984 force-pushed the neutrino-remove-sweeptx branch 3 times, most recently from a0d5cf8 to bcae6bf Compare November 17, 2023 15:26

ziggie1984 requested a review from yyforyongyu November 18, 2023 08:56

yyforyongyu reviewed Nov 20, 2023

View reviewed changes

sweep/sweeper.go Outdated Show resolved Hide resolved

sweep/sweeper.go Show resolved Hide resolved

docs/release-notes/release-notes-0.18.0.md Outdated Show resolved Hide resolved

itest/lnd_onchain_test.go Show resolved Hide resolved

ziggie1984 force-pushed the neutrino-remove-sweeptx branch from bcae6bf to 74dbee7 Compare November 20, 2023 18:56

ziggie1984 requested a review from yyforyongyu November 20, 2023 21:26

ziggie1984 force-pushed the neutrino-remove-sweeptx branch from 74dbee7 to d13aff1 Compare November 21, 2023 08:19

yyforyongyu approved these changes Nov 21, 2023

View reviewed changes

docs/code_formatting_rules.md Show resolved Hide resolved

guggero reviewed Nov 28, 2023

View reviewed changes

ziggie1984 force-pushed the neutrino-remove-sweeptx branch 3 times, most recently from 9ec90ee to f6f1c03 Compare November 29, 2023 13:47

ziggie1984 requested a review from guggero November 29, 2023 13:52

ziggie1984 force-pushed the neutrino-remove-sweeptx branch 2 times, most recently from f7556b9 to 8123112 Compare November 29, 2023 14:30

guggero reviewed Dec 12, 2023

View reviewed changes

ziggie1984 added 6 commits December 12, 2023 13:11

itest: add an itest for the removal of a tx.

1ca3732

lncli: add new removetx cmd.

7c09cc3

This new command calls the new rpc endpoint RemoveTransaction.

docs: add formatting rule for function calls.

58a5dcd

docs: add release-notes

043153f

ziggie1984 force-pushed the neutrino-remove-sweeptx branch from 8123112 to 043153f Compare December 12, 2023 13:19

ziggie1984 commented Dec 12, 2023

View reviewed changes

guggero self-requested a review December 12, 2023 14:40

guggero approved these changes Dec 12, 2023

View reviewed changes

guggero merged commit f2d48c3 into lightningnetwork:master Dec 12, 2023
24 of 25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

neutrino remove sweeptx #7800

neutrino remove sweeptx #7800

ziggie1984 commented Jul 1, 2023 •

edited

Loading

ziggie1984 commented Jul 1, 2023 •

edited

Loading

guggero left a comment

Roasbeef commented Jul 11, 2023

ziggie1984 commented Jul 11, 2023 •

edited

Loading

ziggie1984 commented Aug 9, 2023 •

edited

Loading

ziggie1984 commented Aug 9, 2023

ziggie1984 commented Aug 10, 2023

Roasbeef commented Oct 3, 2023

lightninglabs-deploy commented Oct 31, 2023

ziggie1984 commented Oct 31, 2023

ziggie1984 commented Nov 7, 2023 •

edited

Loading

yyforyongyu left a comment

yyforyongyu left a comment

ziggie1984 commented Nov 20, 2023

yyforyongyu left a comment

guggero left a comment

guggero left a comment

guggero Dec 12, 2023

ziggie1984 Dec 12, 2023 •

edited

Loading

ziggie1984 Dec 12, 2023

guggero Dec 12, 2023

ziggie1984 Dec 12, 2023

guggero Dec 12, 2023

ziggie1984 Dec 13, 2023

guggero left a comment

	// Send another UTXO if this is a neutrino backend. When sweeping
	// anchors, there are two transactions created, `local_sweep_tx` for
	// sweeping Alice's anchor on the local commitment, `remote_sweep_tx`
	// for sweeping her anchor on the remote commitment. Whenever the force
	// close transaction is published, Alice will always create these two
	// transactions to sweep her anchor.
	// On the other hand, when creating the sweep txes, the anchor itself
	// is not able to cover the fee, so another wallet UTXO is needed. In
	// our test case, there's a change output that can be used from the
	// above funding process. And it's used by both sweep txes - when `lnd`
	// happens to create the `remote_sweep_tx` first, it will receive an
	// error since its parent tx, the remote commitment, is not known,
	// hence freeing the change output to be used by `local_sweep_tx`.
	// For neutrino client, however, it will consider the transaction which
	// sweeps the remote anchor as an orphan tx, and it will neither send
	// it to the mempool nor return an error to free the change output.
	// Thus, if the change output is already used in `remote_sweep_tx`, we
	// won't have UTXO to create `local_sweep_tx`.
	//
	// NOTE: the order of the sweep requests for the two anchors cannot be
	// guaranteed. If the sweeper happens to sweep the remote anchor first,
	// then the test won't pass without the extra UTXO, which is the source
	// of the flakeness.
	//
	// TODO(yy): make a RPC server for sweeper so we can explicitly check
	// and control its state.
	if ht.IsNeutrinoBackend() {
	ht.FundCoins(anchorFeeBuffer, alice)
	}

neutrino remove sweeptx #7800

neutrino remove sweeptx #7800

Conversation

ziggie1984 commented Jul 1, 2023 • edited Loading

Change Description

Steps to Test

Pull Request Checklist

Testing

Code Style and Documentation

ziggie1984 commented Jul 1, 2023 • edited Loading

guggero left a comment

Choose a reason for hiding this comment

Roasbeef commented Jul 11, 2023

ziggie1984 commented Jul 11, 2023 • edited Loading

ziggie1984 commented Aug 9, 2023 • edited Loading

ziggie1984 commented Aug 9, 2023

ziggie1984 commented Aug 10, 2023

Roasbeef commented Oct 3, 2023

lightninglabs-deploy commented Oct 31, 2023

ziggie1984 commented Oct 31, 2023

ziggie1984 commented Nov 7, 2023 • edited Loading

yyforyongyu left a comment

Choose a reason for hiding this comment

yyforyongyu left a comment

Choose a reason for hiding this comment

ziggie1984 commented Nov 20, 2023

yyforyongyu left a comment

Choose a reason for hiding this comment

guggero left a comment

Choose a reason for hiding this comment

guggero left a comment

Choose a reason for hiding this comment

guggero Dec 12, 2023

Choose a reason for hiding this comment

ziggie1984 Dec 12, 2023 • edited Loading

Choose a reason for hiding this comment

ziggie1984 Dec 12, 2023

Choose a reason for hiding this comment

guggero Dec 12, 2023

Choose a reason for hiding this comment

ziggie1984 Dec 12, 2023

Choose a reason for hiding this comment

guggero Dec 12, 2023

Choose a reason for hiding this comment

ziggie1984 Dec 13, 2023

Choose a reason for hiding this comment

guggero left a comment

Choose a reason for hiding this comment

ziggie1984 commented Jul 1, 2023 •

edited

Loading

ziggie1984 commented Jul 1, 2023 •

edited

Loading

ziggie1984 commented Jul 11, 2023 •

edited

Loading

ziggie1984 commented Aug 9, 2023 •

edited

Loading

ziggie1984 commented Nov 7, 2023 •

edited

Loading

ziggie1984 Dec 12, 2023 •

edited

Loading