NBXplorer stuck syncing meaning node is stuck on 99%

[cantontowerstation]

Issue arose after server was stopped for an ip change. Everything is updated and has been restarted several times. Server is a VPS running Debian 11 Bullseye with 2 CPU and 8GB RAM.

Web UI shows:

Your nodes are synching... 99%
NBXplorer headers height: 698544
The node is synchronized (Height: 698544)
NBXplorer is synchronizing... (Height: 656552)

NBXplorer docker container logs show this loop:

2021-09-01T13:18:25.236639224Z info: Explorer:       BTC: Starting scan at block 656552
2021-09-01T13:18:25.299181330Z info: Events:         BTC: Node state changed: NotStarted => NBXplorerSynching
2021-09-01T13:19:55.239302182Z info: Explorer:       BTC: Block download seems to hang, let's reconnect to this node
2021-09-01T13:19:55.239526440Z info: Events:         BTC: Node state changed: NBXplorerSynching => NotStarted
2021-09-01T13:19:55.239567758Z info: Configuration:  BTC: Testing RPC connection to http://bitcoind:43782/
2021-09-01T13:19:55.239578903Z info: Explorer:       BTC: Closed connection with node
2021-09-01T13:19:55.259241395Z info: Configuration:  BTC: RPC connection successful
2021-09-01T13:19:55.269099617Z info: Configuration:  BTC: Full node version detected: 210100
2021-09-01T13:19:55.465209467Z info: Configuration:  BTC: Loading chain from cache...
2021-09-01T13:19:55.944958014Z info: Configuration:  BTC: Height: 698545
2021-09-01T13:19:55.949131711Z info: Configuration:  BTC: Trying to connect via the P2P protocol to trusted node (bitcoind:39388)...
2021-09-01T13:19:55.954172946Z info: Explorer:       BTC: TCP Connection succeed, handshaking...
2021-09-01T13:19:55.957912623Z info: Explorer:       BTC: Handshaked
2021-09-01T13:19:55.958979905Z info: Configuration:  BTC: Loading chain from node
2021-09-01T13:19:55.959492464Z info: Explorer:       BTC: Loading chain...
2021-09-01T13:19:55.961523917Z info: Explorer:       BTC: Chain loaded
2021-09-01T13:19:55.968453943Z info: Explorer:       BTC: NBXplorer is correctly whitelisted by the node
2021-09-01T13:19:56.156798245Z info: Configuration:  BTC: Height: 698545
2021-09-01T13:19:56.159204762Z info: Explorer:       BTC: Starting scan at block 656552
2021-09-01T13:19:56.162822437Z info: Events:         BTC: Node state changed: NotStarted => NBXplorerSynching

Let me know what other info would be helpful.

[cantontowerstation]

This was due to me using a pruned node and my server being down for enough time that NBXPlorer was trying to sync blocks that were no longer part of the pruned blocks in bitcoin.

To resolve I had to set environment variables which told NBXPlorer where to sync from and also to rescan.

Firstly I had to find out what block to start from (all commands run from the lcoation of btcpayServer-docker directory where all the .sh files are located)
To check you have a pruned node:
echo $BTCPAYGEN_ADDITIONAL_FRAGMENTS
It should return something like opt-save-storage with additional letters representing how much info is kept on pruning

By the way, to check other variables set you can do:
printenv

Find out the current pruned height:
bitcoin-cli.sh getblockchain info

In the info returned, look for pruned height
Make a note of the block number

Now set the following env vars:
export NBXPLORER_BTCSTARTHEIGHT=<PRUNED HEIGHT YOU JUST FOUND OUT>
export NBXPLORER_BTCRESCAN=1 (this tells NBXPlorer to rescan so that the changed starting height takes effect - the value 1 just means true in this case)

There were 3 different ways I found of setting the variables. I am not sure which method worked for me, so perhaps try them all until it works.

1. Just export the variables by doing:
   export NBXPLORER_BTCSTARTHEIGHT=<PRUNED HEIGHT> and
   export NBXPLORER_BTCRESCAN=1

2. Add the same env vars in this file, in the section entitled nbxplorer and where you can see other environment variables beginning with NBXPlorer_:
   <PATH TO>/btcpayserver-docker/Generated/docker-compose.generated.yml

NBXPLORER_BTCSTARTHEIGHT: <PRUNED HEIGHT>
NBXPLORER_BTCRESCAN: 1

Note the use of colons : not equaal signs =

3. Add the same env vars in this file:
   <PATH TO>/.env
   In my case, the file was located in the directory above the btcpayserver-docker directory

NBXPLORER_BTCSTARTHEIGHT=<PRUNED HEIGHT>
NBXPLORER_BTCRESCAN=1

Then restart BTCPayServer. It was suggested in other threads to stop and start using these commands:
btcpay-down.sh
and then
btcpay-up.sh
If this doesn't work, you could try rebooting the server.

You can check that NBXPlorer is now syncing from the correct block, using this command:
docker logs -f --timestamps generated_nbxplorer_1
This gives real time log updates, if you prefer just the last 100 entries you could do:
docker logs --tail 100 generated_nbxplorer_1

This command worked for me, because the correct name for the NBXPlorer container was generated_nbxplorer_1. If this doesn't work, try finding out if the container name is correct, using:
docker ps

Hopefully that works, then you have to wait ages for it to sync. But it should now fully sync.

I then got rid of the environment variables by deleting the entries I set above in the appropriate files and using the unset command to get rid of the system variables:
unset NBXPLORER_BTCSTARTHEIGHT
unset NBXPLORER_BTCRESCAN

and then restarted everything for good measure:
btcpay-down.sh
then
btcpay-up.sh

Hope that helps.

For reference, these were some related issues that helped me to resolve this.

Ceased to work
#126

BTC node state changing back and forth betweenNBXplorerSynching and notstarted
btcpayserver/btcpayserver#2134

NBXplorer synchronization hanging
btcpayserver/btcpayserver#2722

How to insure tracking from a specific point in time or startheight
dgarage/NBXplorer#15

Using btc.startheight on a block lower than the pruned node
dgarage/NBXplorer#252

[IT4Bitcoin]

This was due to me using a pruned node and my server being down for enough time that NBXPlorer was trying to sync blocks that were no longer part of the pruned blocks in bitcoin.

To resolve I had to set environment variables which told NBXPlorer where to sync from and also to rescan.

Firstly I had to find out what block to start from (all commands run from the lcoation of btcpayServer-docker directory where all the .sh files are located) To check you have a pruned node: echo $BTCPAYGEN_ADDITIONAL_FRAGMENTS It should return something like opt-save-storage with additional letters representing how much info is kept on pruning

By the way, to check other variables set you can do: printenv

Find out the current pruned height: bitcoin-cli.sh getblockchain info

In the info returned, look for pruned height Make a note of the block number

Now set the following env vars: export NBXPLORER_BTCSTARTHEIGHT=<PRUNED HEIGHT YOU JUST FOUND OUT> export NBXPLORER_BTCRESCAN=1 (this tells NBXPlorer to rescan so that the changed starting height takes effect - the value 1 just means true in this case)

There were 3 different ways I found of setting the variables. I am not sure which method worked for me, so perhaps try them all until it works.

1. Just `export` the variables by doing:
   `export NBXPLORER_BTCSTARTHEIGHT=<PRUNED HEIGHT>` and
   `export NBXPLORER_BTCRESCAN=1`

2. Add the same env vars in this file, in the section entitled `nbxplorer` and where you can see other environment variables beginning with `NBXPlorer_`:
   `<PATH TO>/btcpayserver-docker/Generated/docker-compose.generated.yml`

NBXPLORER_BTCSTARTHEIGHT: <PRUNED HEIGHT>
NBXPLORER_BTCRESCAN: 1

Note the use of colons : not equaal signs =

3. Add the same env vars in this file:
   `<PATH TO>/.env`
   In my case, the file was located in the directory above the `btcpayserver-docker` directory

NBXPLORER_BTCSTARTHEIGHT=<PRUNED HEIGHT>
NBXPLORER_BTCRESCAN=1

Then restart BTCPayServer. It was suggested in other threads to stop and start using these commands: btcpay-down.sh and then btcpay-up.sh If this doesn't work, you could try rebooting the server.

You can check that NBXPlorer is now syncing from the correct block, using this command: docker logs -f --timestamps generated_nbxplorer_1 This gives real time log updates, if you prefer just the last 100 entries you could do: docker logs --tail 100 generated_nbxplorer_1

This command worked for me, because the correct name for the NBXPlorer container was generated_nbxplorer_1. If this doesn't work, try finding out if the container name is correct, using: docker ps

Hopefully that works, then you have to wait ages for it to sync. But it should now fully sync.

I then got rid of the environment variables by deleting the entries I set above in the appropriate files and using the unset command to get rid of the system variables: unset NBXPLORER_BTCSTARTHEIGHT unset NBXPLORER_BTCRESCAN

and then restarted everything for good measure: btcpay-down.sh then btcpay-up.sh

Hope that helps.

For reference, these were some related issues that helped me to resolve this.

Ceased to work #126

BTC node state changing back and forth betweenNBXplorerSynching and notstarted btcpayserver/btcpayserver#2134

NBXplorer synchronization hanging btcpayserver/btcpayserver#2722

How to insure tracking from a specific point in time or startheight dgarage/NBXplorer#15

Using btc.startheight on a block lower than the pruned node dgarage/NBXplorer#252

This totally worked for me. Thanks! :D

Only one correction, the command is "bitcoin-cli.sh getblockchaininfo" and not "bitcoin-cli.sh getblockchain info", just in case someone got stuck there

[LouisWayne]

Thanks for sharing this.
For me, <PATH TO>/btcpayserver-docker/Generated/docker-compose.generated.yml was the key to fix the issue.

Thank you!

[sebastianwestberg]

Thank you! I had to do this too on my fresh setup. 🙏

[Delitants]

Same problem, it was down jus a few hours and now it's stuck at 99%.

Find out the current pruned height:
bitcoin-cli.sh getblockchain info

root@btc:~/btcpayserver-docker# ./bitcoin-cli.sh getblockchain info
error code: -32601
error message:
Method not found

Great. Now what?

[IT4Bitcoin]

Same problem, it was down jus a few hours and now it's stuck at 99%.

Find out the current pruned height:
bitcoin-cli.sh getblockchain info

root@btc:~/btcpayserver-docker# ./bitcoin-cli.sh getblockchain info error code: -32601 error message: Method not found

Great. Now what?

the command is "bitcoin-cli.sh getblockchaininfo" and not "bitcoin-cli.sh getblockchain info"
There was a typo in The posted Solution
:)

[Delitants]

Same problem, it was down jus a few hours and now it's stuck at 99%.

Find out the current pruned height:
bitcoin-cli.sh getblockchain info

root@btc:~/btcpayserver-docker# ./bitcoin-cli.sh getblockchain info error code: -32601 error message: Method not found
Great. Now what?

the command is "bitcoin-cli.sh getblockchaininfo" and not "bitcoin-cli.sh getblockchain info" There was a typo in The posted Solution :)

Thanks, I just used NBXPLORER_BTCRESCAN=1 and nothing else, restarted and it fixed it.