Error on Bech32m format introduced in BIP 350 - invalid bech32 checksum variant found Bech32 when Bech32m was expected

[lukasdll]

JsonRpc(Json(Error("invalid bech32 checksum variant found Bech32 when Bech32m was expected", line: 1, column: 77318)))

This error is related to an incorrect Bech32 checksum variant in the JSON-RPC response. The error message indicates that a Bech32 checksum variant was found, but a Bech32m checksum variant was expected.

This issue is most likely caused by an inconsistency between the Bitcoin node (or other service) and the Bitcoin library Bitcoincore_rpc.

The Bech32m format was introduced in BIP 350 to address some limitations of the original Bech32 format and is used for native segregated witness (SegWit) v1+ addresses.

The transaction below is an example that should allow to reproduce the error, it is one of the new Wasabi transactions they introduced in their last update.

https://mempool.space/tx/371a5a5c3f4344e8f6d49395bfa3f76fe38e5a25db10640a5523c0377a825015

[apoelstra]

Can you provide more information? What function from this library were you calling? Is the library outputting data with the wrong checksum or trying to read data with the wrong checksum?

[lukasdll]

Certainly.

The functions being called from the bitcoincore_rpc library when the error occurs are get_raw_transaction, get_raw_transaction_info, and get_tx_out.

Here's a minimal code snippet to reproduce the error.

Error: JSON-RPC error: JSON decode error: invalid bech32 checksum variant found Bech32 when Bech32m was expected at line 1 column 77318

cargo.toml

[package]
name = "bitcoincore-rpc-example"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
bitcoincore-rpc = "0.16.0"
bitcoin = "0.30.0"

main.rs

use bitcoincore_rpc::{Auth, Client, RpcApi};

fn main() {
    // Hard-code the transaction hash
    let txid = "371a5a5c3f4344e8f6d49395bfa3f76fe38e5a25db10640a5523c0377a825015".parse().unwrap();
    
    // Replace with the Bitcoin node IP, PORT, and credentials if any
    let rpc_auth = Auth::UserPass("username".to_string(), "password".to_string());
    let rpc_client = Client::new("http://<IP>:<PORT>", rpc_auth).unwrap();

    // Get the input information and print it
    match get_input_info(&rpc_client, &txid) {
        Ok(inputs) => {
            println!("inputs len: {}", inputs.len());
            for (idx, input) in inputs.iter().enumerate() {
                println!("Input #{}:", idx);
                println!("  Address: {}", input.0);
                println!("  Value: {} satoshis", input.1);
            }
        }
        Err(e) => eprintln!("Error: {}", e),
    }
}

fn get_input_info(
    rpc_client: &Client,
    txid: &bitcoincore_rpc::bitcoin::Txid,
) -> Result<Vec<(String, f64)>, Box<dyn std::error::Error>> {
    // Get the raw transaction

    // Iterate through transaction inputs and collect input information
    let mut inputs_info = Vec::new();
    let tx = rpc_client.get_raw_transaction(txid, None)?;
    let inputs = tx.input.clone();
    for input in inputs {
        let prev_tx = rpc_client.get_raw_transaction_info(&input.previous_output.txid, None)?;
        let value = prev_tx.vout[input.previous_output.vout as usize].value.to_btc();
        let prev_output = rpc_client.get_tx_out(&input.previous_output.txid, input.previous_output.vout, None)?;
        let tx_out =  prev_output.clone().unwrap();
        let address = tx_out.script_pub_key.addresses.as_ref().and_then(|addrs| addrs.get(0)).unwrap().to_string();
        println!("{} BTC - {}", value, address);
        inputs_info.push((address.to_owned().to_string(), value));
    }

    // Return the input information
    Ok(inputs_info)
}

The error specifically occurs during address parsing, likely in this line:

let prev_tx = rpc_client.get_raw_transaction_info(&input.previous_output.txid, None)?;

The library is trying to read data with the wrong checksum format, as it seems to expect a Bech32 checksum but encounters a Bech32m checksum instead. This suggests that the library may not yet fully support the Bech32m format.

[lukasdll]

Upon investigating the root of the problem, it becomes clear that bitcoincore_rpc is not responsible for decoding addresses; instead, it relies on bitcoincore_rpc_json, that relies on rust_bitcoin (specifically bitcoin = { version = "0.29.0", features = ["serde"]}). It might be worthwhile for bitcoincore_rpc_json to update the dependency version to bitcoin 0.30.0, or if that doesn't resolve the issue, the error may need to be reported to the rust_bitcoin maintainer.

This is the diff between 0.29.0, and 0.30.0:
rust-bitcoin/rust-bitcoin@0.29.0...bitcoin-0.30.0

Your feedback on the above would be appreciated Andrew Poelstra

[lukasdll]

Update.

I managed to make the code work, but using the method get_raw_transaction instead of get_raw_transaction_info.

It is still unclear to me why get_raw_transaction_info produces that error. I have had the intuition that it should provide the same as get_raw_transaction, but with additional details.

I add below the corrected code in case it is of interest to others.

use bitcoincore_rpc::{Auth, Client, RpcApi};

fn main() {
    // Hard-code the transaction hash
    let txid = "371a5a5c3f4344e8f6d49395bfa3f76fe38e5a25db10640a5523c0377a825015".parse().unwrap();

    // Replace with the Bitcoin node IP, PORT, and credentials if any
    let rpc_auth = Auth::UserPass("username".to_string(), "password".to_string());
    let rpc_client = Client::new("http://<IP>:<PORT>", rpc_auth).unwrap();

    // Get the input information and print it
    match get_input_info(&rpc_client, &txid) {
        Ok(inputs) => {
            println!("inputs len: {}", inputs.len());
            for (idx, input) in inputs.iter().enumerate() {
                println!("Input #{}:", idx);
                println!("  Address: {}", input.0);
                println!("  Value: {} BTC", input.1);
            }
        }
        Err(e) => eprintln!("Error: {}", e),
    }
}

fn get_input_info(
    rpc_client: &Client,
    txid: &bitcoincore_rpc::bitcoin::Txid,
) -> Result<Vec<(String, f64)>, Box<dyn std::error::Error>> {
    let mut inputs_info = Vec::new();
    let tx = rpc_client.get_raw_transaction(txid, None)?; 
    let inputs = tx.input.clone();
    
    for input in inputs {
        let prev_tx = rpc_client.get_raw_transaction(&input.previous_output.txid, None)?;
        let prev_output = &prev_tx.output[input.previous_output.vout as usize];
        let value = prev_output.value as f64 / 10e8;
        let address = bitcoin::Address::from_script(&prev_output.script_pubkey, bitcoin::Network::Bitcoin)?.to_string();
        inputs_info.push((address.to_owned().to_string(), value));
    }

    Ok(inputs_info)
}

[apoelstra]

the error may need to be reported to the rust_bitcoin maintainer.

I am the rust-bitcoin maintainer :). I will deal with it upstream once I understand the issue. Thanks very much for the specific reproduction!

I managed to make the code work, but using the method get_raw_transaction instead of get_raw_transaction_info.

This is surprising to me -- both methods should ultimately hand off to rust-bitcoin, which shouldn't have any idea what RPC the request is coming from, and fail in the same way.

[apoelstra]

@lukasdll I am not able to reproduce the issue, because the call to gettxout on that transaction fails, since the inputs of the transaction are already spent (by the transaction itself). I tried modifying your code to scan through the entire mempool -- same issue.

So I changed your code to call gettxout on every input from the previous transactions rather than just the spent one, and that failed because the addresses field was blank, since this field was deprecated and removed 2 years ago in Bitcoin Core 22.

So I tried changing that to use the new address field and found that it was only added in https://github.com/rust-bitcoin/rust-bitcoincore-rpc/263 and we have not had a release since then because the maintainer of this crate has been largely unavailable.

So I tried changing your Cargo.toml to use the current master of this repo rather than the published version, and the bug does not appear.

[apoelstra]

@lukasdll what RPC server are you trying to connect to? Given the advanced age of the addresses field and the fact that gettxout is working on old transactions, I guess this is not a Bitcoin Core node?

My suspicion that the other end is broken, not rust-bitcoin.

[lukasdll]

I am the rust-bitcoin maintainer :). I will deal with it upstream once I understand the issue. Thanks very much for the specific reproduction!

All my respect for your work honestly. I'm extremely grateful to people like you maintaining open source projects.

So I changed your code to call gettxout on every input from the previous transactions rather than just the spent one, and that failed because the addresses field was blank, since this field was deprecated and removed 2 years ago in Bitcoin Core 22.

The client is connecting to a Bitcoin Core node, so I checked the version and it is still Bitcoin Core 21, which is probably why I could use the address field. I presume backwards compatibility has certain limitations and I should probably update that node :)

Your suspicions are perfectly right, and accurate.

[apoelstra]

Ok, thanks! I'll try with a Core 21 node then -- I don't think Core would've ever gotten bech32m and bech32 mixed up, so now I'm back to thinking the bug is with rust-bitcoin.

[apoelstra]

@lukasdll sorry, even with Bitcoin Core 0.21 I get the same results -- gettxout will not show me spent outputs and when I modify your program to trawl through the mempool for unspent ones, I don't get any errors.

Is there anything special in your bitcoin.conf?

[lukasdll]

Are you sure that you couldn't reproduce the error replacing get_raw_transaction() with get_raw_transaction_info() on the last version of the code I sent above, and connecting to a v 0.21 node?

I made a mistake on the first version of the simplified code, as you very well pointed out, gettxout made no sense because the outputs were spent already. The second version corrected that, but used get_raw_transaction(), which should be replaced with get_raw_transaction_inf() to reproduce the error.

Also, I would suggest using this transaction: 371a5a5c3f4344e8f6d49395bfa3f76fe38e5a25db10640a5523c0377a825015

I don't think there is anything special in my bitcoin.conf:

daemon=1
txindex=1
rpcuser=<username>
rpcpassword=<password>
rpctimeout=50
rpcallowip=<ip/range>
rpcbint=<ip/port>
listen=1
server=1
rpcport=8332

[apoelstra]

Are you saying that the second code, which you described as "corrected" and that you "got it to work", is actually code that exbhibits the bug? :)

I will try it later today.

[lukasdll]

Sorry, I've been a bit messy on my last messages, please give me the opportunity to catch up.

Calling get_raw_transaction_info() is enough, and below I send the simplest code that allows me to reproduce the error.

The transaction characteristics can be observed using an explorer: https://blockstream.info/tx/062ba7a58cc79eaf3275a3552ef9067e8bc4c3b3b12db213775f3965ab4dd83b

use bitcoincore_rpc::{Auth, Client, RpcApi};

fn main() {
    // Hard-code the transaction hash
    let txid = "062ba7a58cc79eaf3275a3552ef9067e8bc4c3b3b12db213775f3965ab4dd83b".parse().unwrap();

    // Replace IP to correspond with the node endpoint, and set username, and password or make rpc_auth = Auth::None;
    let rpc_auth = Auth::UserPass(<USERNAME>, <PASSWORD>);
    let rpc_client = Client::new("http://<IP>:8332", rpc_auth).unwrap();
    
    let result = rpc_client.get_raw_transaction_info(&txid, None); // <== Returns Err.
    println!("{:?}", result);
}

And those are the dependencies:

[package]
name = "bitcoincore-rpc-example"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
bitcoincore-rpc = "0.16.0"
bitcoin = "0.29.0"

Error message: Err(JsonRpc(Json(Error("invalid bech32 checksum variant found Bech32 when Bech32m was expected", line: 1, column: 77318))))

Proof : https://imgur.com/a/Si5M4ty

I managed to "avoid" the error, by not using the method get_raw_transaction_info(), and call instead get_raw_transaction(), which works without any problem, however, I had to do some extra lifting afterwards to reach the data I needed.

[apoelstra]

Your proof notwithstanding, I cannot reproduce this even with 0.21, using exactly the code you just pasted. It works fine on my end.

[lukasdll]

That is, extremely weird. Do I need to checksum the Bitcoin Core node? I'm pretty sure I did not mess with that at all though.
The server is just a raw Ubuntu 22.04 server.

[lukasdll]

Could it be due to the length of the transaction, that gets truncated at some point? That transaction is particularly large.

[apoelstra]

Maaaaybe. That seems really unlikely -- there is only a tiny tiny chance that a malformed address would appear to be bech32m when it was supposed to be bech32, or vice-versa.

What I suspect is that somehow the node is really really old and doesn't support bech32m. But I dunno, no explanation really seems to make sense to me.

[lukasdll]

I could perhaps parse all transactions in a block, and do stats on how many cause an error. If all of them, or only those with certain characteristics. I could also sync a new Bitcoin Core node, up to date, or with a specific version, and observe whether calling the same code causes the error or not. Let me know what would you recommend to pin point the source of the problem.

[apoelstra]

I think you should use tcpdump -A to find the exact data that the bitcoind is sending over RPC prior to the crash. Then we can at least tell whether the problem is bitcoind's fault or not.