feat: offchain utxo notifications

[dan-da]

Overview

Implements support for off-chain utxo notifications in neptune-core and neptune-cli.

Example Usage (neptune-cli)

bob:

$ neptune-cli next-receiving-address > bob.address

alice:

$ neptune-cli send `cat bob.address` 15 0.1 bob --unowned-utxo-notify-method off-chain-serialized

*** Utxo Transfer Files ***

wrote /tmp/demo/.alice/neptune-core/regtest/utxo-transfer/bob/nolgar1k3tawld...vj487mwx-15.json

*** Important - Read or risk losing funds ***


1 transaction outputs were each written to individual files for off-chain transfer.

-- Sender Instructions --

You must transfer each file to the corresponding recipient for claiming or they will never be able to claim the funds.

You should also provide them the following recipient instructions.

-- Recipient Instructions --

run `neptune-cli claim-utxo file <file>` or use equivalent claim functionality of your chosen wallet software.

Send completed. Tx Digest: 8b634f77ce8edf764769c1b9d8eaf2e71bf3b10572b27f7af53202d6a0dd3119d983eac092eb1f82

bob:

$ neptune-cli claim_utxo file nolgar1k3tawld...vj487mwx-15.json

Success.  1 Utxo Transfer was imported.

Demo

Utxo transfer file location, naming, and serialization

neptune-cli generates one file per output notification serialized via json pretty-print. It is intended to be human and machine readable.

The file path is <neptune-data-dir>/utxo-transfer/<recipient>/<basename>.json

where:

• neptune-data-dir is retrieved from neptune-core via RPC.
• recipient is a label provided by --recipient arg, or "anonymous" by default, or "self" if the output is owned by "our" wallet.
• basename is
  • nolga<pubkeyfirst8char>...<pubkeylast8char> (for generation addresses)
  • <recipient_id> (for symmetric keys)

note: recipient is a local label that never leaves the local device.

note: recipient id is used for symmetric keys to avoid exposing any secret key material.

Utxo transfer file data

The data consists of single UtxoTransferEntry, defined as:

pub struct UtxoTransferEntry {
    pub data_format: String,
    pub recipient: String,
    pub amount: String,
    pub utxo_transfer_encrypted: String,
    pub address_info: AddressEnum,
}
pub enum AddressEnum {
    Generation {
        address_abbrev: String,
        address: String,
        receiver_identifier: String,
    },
    Symmetric {
        receiver_identifier: String,
    },
}

The only field necessary for claiming is utxo_transfer_encrypted. Everything else is for informational purposes only, with the intent to help the sender route the utxo to the correct recipient. In particular, the address field is useful to match against the address that was originally provided to generate_tx_params().

The recipient label is intentionally NOT included in the file itself or the filename because doing so might expose private information during transit, and also a sender might not wish the recipient to see the sender's private label for the recipient.

UtxoTransfer and UtxoTransferEncrypted

utxo secrets are placed in UtxoTransfer and then encrypted to the recipient's pubkey creating UtxoTransferEncrypted. The latter is then encoded with bech32 into a network-specific String.

pub struct UtxoTransfer {
    pub utxo: Utxo,
    pub sender_randomness: Digest,
}
pub struct UtxoTransferEncrypted {
    /// contains encrypted UtxoTransfer
    pub ciphertext: Vec<BFieldElement>,

    /// enables the receiver to find the matching `SpendingKey` in their wallet.
    pub receiver_identifier: BFieldElement,
}

Claiming with neptune-cli

The neptune-cli claim_utxo command accepts 3 forms of data:

1. file: path to a file containing a single json encoded UtxoTransferEntry
2. json: string containing a single json encoded UtxoTransferEntry
3. raw: string containing bech32 encoded UtxoTransferEncrypted.

(1) is the intended/default usage. (2) and (3) are provided for added convenience or special cases.

claim_utxo rpc

claiming works by:

1. decode bech32 network-specific string into UtxoTransferEncrypted
2. find the known wallet key that corresponds to recipient_id
3. decrypt ciphertext into UtxoTransfer
4. find monitored_utxo and/or expected_utxo that match target utxo.
5. find block in which utxo's parent Tx was confirmed, if any
   5a. if found, get blocks from confirmation to tip.
   5b. create a new monitored utxo and add membership proofs, etc.
6. perform mutations/writes:
   6a. add expected utxo to wallet. (always)
   6b. add monitored utxo to wallet (if confirmed)
   6c. persist wallet.

neptune-core refactors

To make this work smoothly it was necessary to refactor send(), create_transaction() and related APIs. A nice benefit is that send() and create_transaction() now both work for any possible Tx, ie arbitrary lock-scripts and arbitrary Coins. Also the caller specifies all inputs and ouputs, including change output.

To make it easier for the caller to determine input and calculate change, two rpc are available: generate_tx_params() and generate_tx_params_from_tx_outputs(). The former accepts the same params that send() used to (plus 2 more option) and is intended for simple usage when sending native-coin amounts to one or more recipients with ReceivingAddress. The latter is for sending to arbitrary LockScript and Coin, but chooses inputs and generates a change address.

See doc-comments for more description of these and other changes.

Claiming modes

unconfirmed. Claiming can occur before the corresponding tx is confirmed in a block or after. If it occurs before, all that happens is for an ExpectedUtxo to be added to the wallet. Note that this will not be reflected in the wallet balance until the tx is confirmed. See #184.

confirmed. If claiming is performed after the tx is confirmed then additional computation is required. In particular a new MonitoredUtxo must be created and it must have a MembershipProof for every block since the tx was created until the tip. This requires an archival-node. It also must be marked as spent in the unusual case that a newly-claimed utxo has somehow already been spent.

In the confirmed mode, the wallet balance will immediately reflect the claimed funds once claim_utxo() rpc completes.

New unit tests:

• claim_utxo_owned_before_confirmed
• claim_utxo_owned_after_confirmed
• claim_utxo_unowned_before_confirmed
• claim_utxo_unowned_after_confirmed
• validate_insufficient_inputs
• validate_negative_input_amount
• validate_negative_output_amount

Design Decisions

These are some decisions I had to think about, and my rationale. Not written in stone.

1. owned off-chain serialized notifications. For change output or any other "owned" utxo there doesn't seem to be a really compelling use case for offchain serialized notifications. For now I decided to leave them in (a) for completeness, and (b) for the use-case of practicing. When performing any kind of cryptocurrency Tx it is often advisable to practice with your own wallet first, and this provides a way to do that. Counter arguments would be that they add cognitive load, eg to understand the optional send() flags, and also require special handling for symmetric keys.

2. single or multiple utxo entries per utxo-transfer file. It is possible for a tx to have multiple outputs destined for the same receiving address. In which case it would make sense to place them both in the same file. However this usage seems likely to be uncommon and it just seems overall simpler to go with a one-utxo-per-file approach.

3. who writes the utxo-transfer files. It could be that neptune-core writes these files and just provides the client with a path, or that the client writes them. I chose the latter, as it permits an rpc client on a different device to work with the APIs and generally seems more flexible. A drawback is that neptune-cli and neptune-dashboard will each have to write them, but that logic can be shared easily enough.

Unresolved:

1. I'm not entirely happy with naming yet and open to revisiting. In particular:

I'm not sure "utxo-transfer" is quite right though I like it is pretty short and conveys a send/receive action. The UtxoTransfer struct does contain a Utxo struct, but I think it more correct to say the transfer takes place onchain, and this is just a notification of it. But we already have a struct UtxoNotification in neptune-core. Perhaps it should be renamed? Another possibility might be "utxo-secrets".

2. The enum type OffchainSerialized is kinda verbose and is not really consistent with the "utxo-transfer" naming used elsewhere. Though they both refer to the same thing. So it would be good to sync those up somehow.

3. There are presently no unit tests for the case when a utxo has already been spent before it is claimed. That should be impl'd before this PR is merged.

4, There is presently no support for offchain utxo-transfers in neptune-dashboard. I figured it's better to get buy-off on the direction in this PR before tackling tui for the feature.

Doubts / Concerns:

This functionality should be considered highly experimental. It relies on end-users (or 3rd party systems) to transmit secrets and secure data against loss. Failure to do so can result in permanent loss-of-funds. It's one of those "with great power comes great responsibility" kind of things.

I have a concern (a) that people will manage to lose funds with this, and (b) that could create a bad image/perception for neptune-cash.

At the same time, it is an interesting feature/innovation and possibly could help with scaling and offloading data to other systems. I don't think we can predict now all the different ways it could be used. So it may well be worth trying out and we shouldn't let perfect be the enemy of the good.

For Reviewers.

I would suggest the most important things to look at are:

neptune-core:

• WalletState::prepare_claim_utxo_in_block
• WalletState::finalize_claim_utxo_in_block
• UtxoTransfer, UtxoTransferEncrypted
• GlobalState::create_transaction(), generate_tx_params().
• changes to the rpc API.

neptune-cli:

• changes to the send and send-to-many commands.
• format of the json file. (UtxoTransferEntry)

Changes

neptune-core:

• add DataDirectory::utxo_transfer_directory_path()
• make DataDirectory serializable
• derive(clap::ValueEnum) for Network
• improve Network doc-comments for use in neptune-cli help.
• Network::RegTest --> Regtest
• impl Add,AddAssign for BlockHeight
• simplify BlockSelector impl
• add Block::genesis_prev_block_digest()
• add trait BlockchainBlockSelector
• make TxInput serializable
• change TxIput::spending_key to unlock_key
• add type alias TxAddressOutput = (ReceivingAddress, NeptuneCoins)
• split UtxoNotifyMethod into Owned and Unowned variants.
• make UtxoNotification serializable
• add OffChainSerialized to utxo notification enums.
• make TxOutput serializable
• add TxOutput::offchain_serialized()
• add UtxoTransferList::utxo_transfer_iter()
• rename TransactionDetails --> TxParams
• rename NeptuneCoins::one() -> one_nau()
• derive clap::ValueEnum for KeyType
• make SpendingKey serializable
• add ReceivingAddress::bech32m_abbreviated()
• make GenerationSpendingKey serializable
• impl bech32m for SymmetricKey with prefix "nsk"
• add UtxoNotifier::Claim variant
• add struct wallet::UtxoTransfer
• prevent dups in WalletState::add_expected_utxo()
• add WalletState::has_expected_utxo()
• add WalletState::find_monitored_utxo()
• add WalletState::find_known_spending_key_for_receiver_identifier()
• add WalletState::prepare_claim_utxo_in_block()
• add WalletState::finalize_claim_utxo_in_block()
• add WalletStatus::synced_unspent_amount()
• cache tip block in ArchivalState.
• add ArchivalState::canonical_block_stream_asc()
• add ArchivalState::canonical_block_stream_desc()
• impl BlockchainBlockSelector for ArchivalState
• impl BlockchainBlockSelector for BlockchainState
• remove BlockchainArchivalState
• add GlobalStateLock::generate_tx_params()
• add GlobalStateLock::send()
• simplify create_transaction()
• log detailed message when storing block.
• add struct TxOutputMeta
• add shared test method mine_block_to_wallet()
• insert tx into mempool in GlobalStateLock::send() rather than main loop.
• add rpc data_directory()
• add rpc generate_tx_params()
• add rpc generate_tx_params_from_tx_outputs()
• add rpc claim_utxo()
• remove rpc send_to_many()
• add rpc test: claim_utxo_owned_before_confirmed()
• add rpc test: claim_utxo_owned_after_confirmed()
• add rpc test: claim_utxo_unowned_before_confirmed()
• add rpc test: claim_utxo_unowned_after_confirmed()

neptune-cli:

• add doc-comment description of every command for help
• remove redundant tip-header command
• renamed own-receiving-address --> next-receiving-address
• next-receiving-address accepts key-type arg
• send accepts optional params: recipient, unowned-utxo-notify-method, owned-utxo-notify-method
• send-to-many accepts same parameters parsed from outputs list.
• send and send-to-many create utxo-transfer file(s) if necessary and instructions for sender and recipient.
• improved help-text of send, send-to-many. shows variants of enum params.
• send and send-to-many obtain data_dir from neptune-core.
• added claim-utxo command.
• wallet maintenance commands accept optional data_dir
• --network is now command specific.

cargo.toml:

• add clap feature "string"

[aszepieniec]

I imagine that this feature could be used as follows.

• Bob sends the receiving address to Alice via email.
• Alice sends the UTXO to Bob via transaction on the blockchain.
• Alice sends the UtxoTranser (actually, I like "UtxoSecrets" better) to Bob via email attachment.

I'm concerned about Bob's privacy from his email service provider. If I understand correctly, the file itself is encrypted but the file name contains both the amount and Bob's address. If so, it is suboptimal and I would propose to address it in one of two ways.

1. Rather than writing to a file, the CLI outputs the bech32m-encoded ciphertext directly to stdout. It is Alice's responsibility to pipe it into a file, or just copy-paste the output into the email body.
2. Select a different file name. This strategy comes with a new problem whereby Alice somehow needs to keep track of UtxoSecrets she sent and what they were for, and now the file name does not help.

Also, I imagine Bob at some later date could be confused as to whether he already claimed a given UTXO. What happens if he tries to claim it again?