Resetting Testnet: Bitcoin Explorers, Indexers & APIs

[CharlieC3]

Today we discussed the pros/cons of running a Stacks Testnet against a private Bitcoin Regtest network. One of the challenges we're looking to understand is the difficulties and feature-gaps of running a Bitcoin explorer, indexer, and API which other entities and groups in the Stacks ecosystem may rely upon, as many of these services are available on the internet for Bitcoin mainnet (e.g. https://mempool.space/).

Some folks at Hiro, such as myself, @zone117x and @saralab, researched a bunch of different OSS in this field in early 2023. Our research led us to understand that the mempool frontend & backend projects, along with the mempool electrs indexer/API had the best feature-set and alleged performance and reliability at the time.
This is what we're using to power https://mempool.bitcoin.regtest.hiro.so/

To better understand the challenges in running these services and their feature-gaps, I thought it would be useful to provide the research we conducted in early 2023 to facilitate a discussion around this matter in hopes that it helps us make a more informed decision on what the future of the Stacks Testnet network may look like.

It should be noted that:

• Many of the different projects in the Bitcoin explorer/api/indexer space are often forks of each other with usually minor differences between them. So if there are incompatibility issues found in one of our selections, replacing the component with a different project may not resolve any compatibility issues as they may be based on the same code, or may introduce other challenges that the original project accounts for, such as performance or reliability concerns.
• As this research was conducted in early 2023, take it with a grain of salt and when in doubt verify it again as the status for some of these projects may have changed.
• The mempool electrs indexer/API that we chose is not mentioned in the list below, as it was created between the time we finished this research and implemented it. But since it was based on romanz/electrs and Blockstream/electrs which our research favored, it checked all the boxes and made the most sense to use with the mempool frontend & backend projects.

Bitcoin Indexer Comparison (early 2023)

Project	Protocol	Prod-ready	Deployment Scope	Scalability	Metrics support	Community Support	Verdict
romanz/electrs	⚠️ Electrum	❌ Note that this implementation of Electrum server is optimized for personal/small-scale (family/friends) usage. It's a bad idea to run it publicly as it'd expose you to DoS and maybe also other attacks.	⚠️ Hiro would need to build and maintain a Docker image Requires an empty disk to store database locally	❌ Does not scale well; scaling up would require a separate database. This could be done with snapshots potentially, but is time-consuming and increases overhead. Load balancing may return differing data with session persistence.	✅ (Prometheus)	✅ > 500 stars Latest official release: 1/5/23	❌ 1 Intended for small-scale operations
blockstream/electrs	✅ Electrum + Esplora	✅ Used as the backend for the Esplora block explorer powering blockstream.info.	⚠️ Hiro would need to build and maintain a Docker image Requires an empty disk to store database locally	❌ Does not scale well; scaling up would require a separate database. This could be done with snapshots potentially, but is time-consuming and increases overhead. Load balancing may return differing data with session persistence.	✅ (Prometheus)	⚠️ ~200 stars Latest official release: 10/14/18 Latest commit to default branch: 6/22/22 The GH issues page has been hidden; no clear way to submit new issues to track current ones	⚠️ 3 Flexible due to combination of protocols May need to change the storage engine to improve scalability.
coinbase/rosetta-bitcoin	✅ Rosetta	❌ ROSETTA-BITCOIN IS CONSIDERED ALPHA SOFTWARE.	⚠️ Hiro would need to build and maintain a Docker image Requires an empty disk to store database locally	❌ Does not scale well; scaling up would require a separate database. This could be done with snapshots potentially, but is time-consuming and increases overhead. Load balancing may return differing data with session persistence.	❌	⚠️ ~90 stars Latest official release: 2/8/22 Latest commit to default branch: 11/1/22	❌ 1 Coinbase is not running this and has not maintained it for a while
dpc/rust-bitcoin-indexer	✅ Custom (Postgres)	⚠️ An experiment in creating a perfect Bitcoin Indexer ... This started as an educational experiment, but is quite advanced already. ... rust-bitcoin-indexer (at least right now) is just a neat experiment that gave good results and seems to be working quite well.	✅ Hiro would need to build and maintain a Docker image Requires a Postgres database deployed	⚠️ Uses a Postgres db, so the app can theoretically scale horizontally. Since this project is not prod-ready, we’d still need to verify how well the app performs under heavy load.	❌	❌ ~90 stars No official releases/tags Latest commit to default branch: 5/17/21	❌ 1 Recognized as an “experiment”, low dev activity
cculianu/Fulcrum	⚠️ Electrum (Fulcrum fork)	❔ unclear	⚠️ Hiro would need to build and maintain a Docker image Requires an empty disk to store database locally	❌ Does not scale well; scaling up would require a separate database. This could be done with snapshots potentially, but is time-consuming and increases overhead. Load balancing may return differing data with session persistence.	❌	✅ ~160 stars Latest official release: 12/20/22 Active development, issues being resolved regularly	❌ 2 Poor scaling capabilities, Electrum protocol has limitations
spesmilo/electrumx	⚠️ Electrum	❔ unclear	⚠️ Hiro would need to build and maintain a Docker image Requires an empty disk to store database locally	❌ Does not scale well; scaling up would require a separate database. This could be done with snapshots potentially, but is time-consuming and increases overhead. Load balancing may return differing data with session persistence.	❌	⚠️ ~300 stars Latest official release: 12/10/20 Latest commit to default branch: 11/10/22 Despite no official release/tag in a while, there’s still plenty of active development on this project	❌ 2 Poor scaling capabilities, Electrum protocol has limitations
block-core/blockcore-indexer	✅ Blockcore	✅ Used as the backend for the Blockcore.net explorer	✅ Official Docker image exists Requires a MongoDB database deployed	⚠️ Uses MongoDB, so the app can theoretically scale horizontally better than an app using a local directory can. Since this project is used by Blockcore.net, chances are it has some capability to scale. It seems blockcore.net is a personal project or doesn’t receive high traffic. Scaling exploration would need to be carried out	❌	⚠️ 17 stars Latest official release: 12/16/22 Latest commit to default branch: 12/21/22 Seems like a small team actively working on this project; issues getting resolved at a slow pace. If Hiro needs a new feature added to bug fixed, we might need to contribute. Written in C#; Hiro doesn’t have many C# devs besides Matt 🙂	⚠️ 2.5 Low community usage Hiro isn’t historically a C# shop Low confidence in its usage for a low traffic site
bitauth/chaingraph	✅ Custom (GraphQL + Postgres)	❌ Limited BTC support (was designed for BCH)	✅ Official Docker images exist Requires a Postgres database deployed	⚠️ Uses a Postgres db, so the app can theoretically scale horizontally. Since this project is not prod-ready, we’d still need to verify how well the app performs under heavy load. Also has a K8s Helm chart, making is simple to deploy internally.	⚠️ Really limited and custom metrics. Might be able to find a way to stream this into Prometheus	⚠️ 24 stars Latest official release: 5/14/22 Latest commit to default branch: 1/6/23 Despite some active development, issues are not getting resolved at all	❌ 1 Limited/questionable BTC functionality BTC support is unofficial
bitpay/bitcore	✅ Custom (REST)	✅ Used as the backend for the Bitpay.com Explorer	⚠️ Hiro would need to build and maintain a Docker image Requires a MongoDB database deployed Some endpoints may need to be blocked.	✅ Uses MongoDB, and supports specification of the db connection details in the config, so the app can theoretically scale horizontally. Since this project is used by Bitpay.com, chances are it has some capability to scale.	⚠️ Really limited and custom metrics. Might be able to find a way to stream this into Prometheus	✅ 4.5k stars Latest official release: 10/16/22 (npm) 1/2/23 (gh tag) Active development, issues being resolved regularly	⚠️ 3.5 Good dev activity and community support Used in high(ish) traffic Explorer Scales with remote database
mempool/mempool	✅ Custom (REST) Integrates with a backend Electrum or Esplora server to increase functionality.	✅ Used as the backend for the mempool.space	✅ Official Docker images exist May require deployment of an Electrum server for increased functionality. Seems to be compatible with romanz/electrs or cculianu/Fulcrum and Blockstream/electrs. Lots of configuration suggestions for deploying to production.	✅ Uses MariaDB, and supports specification of the db connection details in the config, so the app can theoretically scale horizontally. Since this project is used by mempool.space, chances are it has some capability to scale.	❌	✅ 1.2k stars Latest official release/tag: 7/25/22 Despite no new release in 6 months, there’s active development almost every day against master, issues being resolved regularly	✅ Charlie: 4 Matt: 4 Sarala : 4

Protocol usability

The Electrum protocol has a limitation that makes fetching txs for a given block pretty expensive (see blockchain.transaction.id_from_pos).

This may be a considered a downside for the following projects: romanz/electrs, cculianu/Fulcrum, spesmilo/electrumx.

---

[jcnelson]

Taking notes from the call:

• A strong point against testnet4 is that it still encounters block storms, which makes it unsuitable for hosting Nakamoto (since testnet4 could produce an empty prepare phase)

• Signet may not have block storms, but we need to check. Is it a viable alternative?

  • It definitely has flash blocks though
  • It would require potentially-substantial code changes in the Stacks node to support
  • It still forces users to run a signet bitcoind node and somehow obtain coins for it

• A strong point against hosting a testnet inhouse is that we have to solve the problem of tBTC running out without resetting the testnet.

  • Running a fork of bitcoind where this isn't a problem requires all partners to run our forked code
  • We have to maintain the fork

• Running a testnet Stacks miner requires additional services to operate on an inhouse testnet (but these all seem solvable):

  • It would require us to add a way for them to register their mining key(s) in our hosted bitcoind node, if they don't run one themselves
    • Probably not something we need to do right now, since people can already peer their own bitcoind nodes
  • We would need to solely control block mining
    • Already do this
  • It would require us to host and maintain a tBTC faucet (this is already done!), and defend it from griefers
    • The faucet could be much smaller, and we could make our miners mine less
    • Our stackers should treat the faucet address as their PoX rewards address
    • We should publish the testnet faucet address as a sink, so people can return their testnet BTC
  • We would need to support users peering their regtest bitcoind nodes
  • Slower block times to save BTC?

• Set a schedule for nuking public testnet?

• Write docs for setting up a local "testnet2" so we can iterate and test all of the above

Consensus at the end of the call seems to be to keep our current setup as-is, but iterate on it to make it more robust. No concrete actions are to be taken until January, given the upcoming holiday.

[CharlieC3]

No concrete actions are to be taken until January, given the upcoming holiday

FYI @aldur

[aldur]

Thanks both!

Speaking from the experience of the sBTC launch: it has been very challenging for downstream users (e.g., wallets) to integrate sBTC because of the lack of a stable testnet environment.

We seem to have one working right now (running 3.1.0.0.1-rc2). If that's the case, I'd be OK leaving things as they are through the holidays.

However, if the deployment we have turns out to be brittle and continues creating challenges for builders, then I'd advocate for an immediate reset of the testnet. This is of course just a band-aid, giving builders a way to integrate sBTC now when momentum requires it. We would still need to iterate and make more robust later.

[aldur]

• Set a schedule for nuking public testnet?

Optimizing for simplicity, mid-term this seems to be a good option.

[wileyj]

One other thing that i'd like for us to think about - a contingency plan for the single bitcoin regtest rpc miner.
not a blocker to reset, and i think we're all in agreement that the issues raised in this discussion can be addressed afterwards, but i would also like to add a contingency for the miner specifically - either as a hot/cold standby miner (probably the better option), or we alternate who mines blocks (this would be difficult though, since timing could prove challenging).

i'll dig into some ideas here, the last time i looked there weren't any good options that would allow decentralized miners.