Zebra 1.4.0

Zebra 1.4.0 - 2023-11-07

Zebra's mining RPCs are now available in release builds. Our Docker images are significantly
smaller, because the smaller Zcash verification parameters are now built into the zebrad binary.
Zebra has updated to the shared Rust dependencies from the zcashd 5.7.0 release.

Zebra recovers better from brief network interruptions, and avoids some network and verification
denial of service and performance issues. We have restored our macOS tests in CI, and now support
macOS on a best-effort basis.

We have changed our documentation website URL, and we are considering deprecating some Docker image
tags in release 1.5.0 and later.

Deprecation Warnings

This release has the following deprecation warnings:

Warning: Deprecation of DockerHub Image Tags in a future release

Zebra currently publishes 11 DockerHub tags for each new release.
We want to reduce the number of DockerHub tags we publish in a future minor Zebra release.

Based on usage and user feedback, we could stop publishing:

• The 1 tag, which updates each release until NU6
• The 1.x tag, which updates each patch release until the next minor release
• The 1.x.y tag, which is the same as v1.x.y
• The sha-xxxxxxx tag, which is the same as v1.x.y (for production releases)

We also want to standardise experimental image tags to -experimental, rather than .experimental.

So for release 1.5.0, we might only publish these tags:

• latest
• latest-experimental (a new tag)
• v1.5.0
• v1.5.0-experimental

Please let us know if you need any other tags by opening a GitHub ticket.

We recommend using the latest tag to always get the most recent Zebra release.

Warning: Documentation Website URL Change

We have replaced the API documentation on the doc.zebra.zfnd.org
website with docs.rs. All links have been updated.

Zebra's API documentation can be found on:

• docs.rs, which renders documentation for the
  public API of the latest crate releases;
• doc-internal.zebra.zfnd.org, which renders
  documentation for the internal API on the main branch.

doc.zebra.zfnd.org stopped being updated a few days before this release,
and it will soon be shut down.

Significant Changes

This release contains the following significant changes:

Mining RPCs in Production Builds

Zebra's mining RPCs are now available in release builds (#7740). Any Zebra instance can be used
by a solo miner or mining pool. This stabilises 12 RPCs, including getblocktemplate, submitblock,
getmininginfo, getnetworksolps, [z_]validateaddress and getblocksubsidy. For more information,
read our mining blog post.

Please let us know
if your mining pool needs extra RPC methods or fields.

Zcash Parameters in zebrad Binary

zebrad now bundles zk-SNARK parameters directly into its binary. This increases the binary size
by a few megabytes, but reduces the size of the Docker image by around 600 MB because
the parameters don't contain the Sprout proving key anymore. The zebrad download
command does nothing, so it has been removed.

Previously, parameters were stored by default in these locations:

• ~/.zcash-params (on Linux); or
• ~/Library/Application Support/ZcashParams (on Mac); or
• C:\Users\Username\AppData\Roaming\ZcashParams (on Windows)

If you have upgraded zebrad to 1.4.0 or later, and zcashd to 5.7.0 or later, you can delete the
parameter files in these directories to save approximately 700 MB disk space.

zcashd have deprecated their fetch-params.sh script,
so it can't be used to retry failed downloads in zebrad 1.3.0 and earlier.

We recommend upgrading to the latest Zebra release to avoid download issues in new installs.

macOS Support

macOS x86_64 is now supported on a best-effort basis. macOS builds and some tests run in Zebra's CI.

Security

• Reconnect with peers after brief network interruption (#7853)
• Add outer timeouts for critical network operations to avoid hangs (#7869)
• Set iterator read bounds where possible in DiskDb, to avoid a known RocksDB denial of service issue (#7731, #7732)
• Fix concurrency issues in tree key formats, and CPU usage in genesis tree roots (#7392)

Removed

• Remove the zebrad download command, because it no longer does anything (#7819)

Added

• Enable mining RPCs by default in production builds (#7740)
• Re-enable macOS builds and tests in CI (#7834)
• Make macOS x86_64 a tier 2 supported platform in the docs (#7843)
• Add macOS M1 as a tier 3 supported platform (#7851)

Changed

• Build Sprout and Sapling parameters into the zebrad binary, so a download server isn't needed (#7800, #7844)
• Bump ECC dependencies for zcashd 5.7.0 (#7784)
• Refactor the installation instructions for the s-nomp mining pool software (#7835)

Fixed

• Make the latest Docker tag point to the production build, rather than the build with experimental features (#7817)
• Fix an incorrect consensus-critical ZIP 212 comment (#7774)
• Fix broken links to zebra_network and zebra_state Config structs on doc-internal.zebra.zfnd.org (#7838)

Contributors

Thank you to everyone who contributed to this release, we couldn't make Zebra without you:
@arya2, @gustavovalverde, @mpguerra, @oxarbitrage, @rex4539, @teor2345, @upbqdn, and @vuittont60.

Zebra 1.3.0

This release adds RPC methods for the "Spend before Sync" light wallet feature,
and fixes performance issues and bugs in the mining solution rate RPCs. Progress
bars can now be enabled using a config, please help us test them!

It contains the following updates:

User Testing: Progress Bars

Zebra has progress bars! When progress bars are enabled, you can see Zebra's blocks,
transactions, and peer connections in your terminal. We're asking Zebra users to test this
feature, and give us feedback on the forums.

To show progress bars while running Zebra, add these lines to your zebrad.toml:

[tracing]
progress_bar = "summary"

For more details, including a known issue with time estimates,
read our progress bars blog post.

Security

• Fix database concurrency bugs that could have led to panics or incorrect history tree data (#7590, #7663)

Added

• Zebra's progress bars can now be enabled using a zebrad.toml config (#7615)
• Add missing elasticsearch flag feature to lib docs (#7568)
• Add missing Docker variables and examples (#7552)
• Check database format is valid on startup and shutdown (#7566, #7606). We expect to catch almost all database validity errors in CI (#7602, #7627), so users are unlikely to see them on startup or shutdown.

Spend before Sync Support

• Add state requests and support code for the z_getsubtreesbyindex RPC (#7408, #7734)
• Implement the z_getsubtreesbyindex RPC (#7436)
• Test the z_getsubtreesbyindex RPC (#7515, #7521, #7566, #7514, #7628)
• Format subtree roots in little-endian order (#7466)
• Add note subtree indexes for new and existing blocks (#7437)
• Upgrade subtrees from the tip backwards, for compatibility with wallet syncing (#7531)
• Handle a subtree comparison edge case correctly (#7587)

Changed

• Return errors instead of panicking in methods for Heights (#7591)
• Update tests for compatibility with the ECC's lightwalletd fork (#7349)

Fixed

• Refactor docs for feature flags (#7567)
• Match zcashd's getblockchaininfo capitalisation for NU5 (#7454)
• Fix bugs and performance of getnetworksolps & getnetworkhashps RPCs (#7647)

Contributors

Thank you to everyone who contributed to this release, we couldn't make Zebra without you:
@arya2, @gustavovalverde, @oxarbitrage, @rex4539, @teor2345 and @upbqdn.

Zebra 1.2.0

Zebra 1.2.0 - 2023-09-01

Highlights

This release:

• Starts our work implementing "spend before sync" algorithm for lightwalletd.
• Contains an automatic database upgrade that reduces the size of Zebra's current cached state from approximately 276GB to 244GB. It does so by automatically pruning unneeded note commitment trees from an existing cache. New Zebra instances will also build their cache without these trees.

Breaking Changes

zebrad 1.2.0 cached states are incompatible with previous zebrad versions:

• zebrad 1.2.0 upgrades the cached state format. The new format is incompatible with previous zebrad versions. After upgrading to this Zebra version, don't downgrade to an earlier version.
• When earlier versions try to use states upgraded by zebrad 1.2.0:
  • zebrad versions 1.0.0 and 1.0.1 will respond to some z_gettreestate RPC requests with incorrect empty final_state fields
  • pre-release zebrad versions can panic when verifying shielded transactions, updating the state, or responding to RPC requests

Changed

• Deduplicate note commitment trees stored in the finalized state (#7312, #7379)
• Insert only the first tree in each series of identical trees into finalized state (#7266)
• Our testing framework now uses the ECC lightwalletd fork (#7307). This was needed to start the work of implementing fast spendability. The ECC repo is now the supported implementation in Zebra, documentation was changed to reflect this. (#7427)

Added

• Documentation for mining with Docker (#7179)
• Note tree sizes field to getblock RPC method (#7278)
• Note commitment subtree types to zebra-chain (#7371)
• Note subtree index handling to zebra-state, but we're not writing subtrees to the finalized state yet (#7334)

Fixed

• Log a warning instead of panicking for unused mining configs (#7290)
• Avoid expensive note commitment tree root recalculations in eq() methods (#7386)
• Use the correct state version for databases without a state version file (#7385)
• Avoid temporary failures verifying the first non-finalized block or attempting to fork the chain before the final checkpoint (#6810)
• If a database format change is cancelled, also cancel the format check, and don't mark the database as upgraded (#7442)

Contributors

Thank you to everyone who contributed to this release, we couldn't make Zebra without you:
@arya2, @dconnolly, @gustavovalverde, @mpguerra, @oxarbitrage, @teor2345 and @upbqdn

Zebra 1.1.0

This release adds new mempool metrics, fixes panics when cancelling tasks on shutdown, detects subcommand name typos on the command-line, and improves the usability of Zebra's Docker images (particularly for mining).

Breaking Changes

• Zebra now detects subcommand name typos on the command-line. If you want to give Zebra a list of tracing filters, use zebrad start --filters debug,... (#7056)

Security

• Avoid initiating outbound handshakes with IPs for which Zebra already has an active peer (#7029)
• Rate-limit inbound connections per IP (#7041)

Added

• Metrics tracking mempool actions and size bucketed by weight (#7019) by @str4d
• Legacy state format compatibility layer and version bumps for ECC dependencies to match zcashd 5.6.0 (#7053)
• Framework for upcoming in-place database format upgrades (#7031)

Changed

• Deduplicate note commitment trees in non-finalized state (#7218, #7239)

Fixed

• Enable miners running Zebra with Docker to set their address for mining rewards (#7178)
• Use default RPC port when running Zebra with Docker (#7177, #7162)
• Stop panicking on async task cancellation on shutdown in network and state futures (#7219)
• Remove redundant startup logs, fix progress bar number, order, and wording (#7087)
• Organize Docker ENV and ARG values based on their usage (#7200)
• Avoid blocking threads by awaiting proof verification results from rayon in async context (#6887)

Contributors

Thank you to everyone who contributed to this release, we couldn't make Zebra without you:
@arya2, @gustavovalverde, @mpguerra, @oxarbitrage, @str4d, @teor2345 and @upbqdn

Zebra 1.0.1

Zebra 1.0.1 - 2023-07-03

Zebra's first patch release fixes multiple peer connection security issues and panics. It also significantly reduces Zebra's CPU usage. We recommend that all users upgrade to Zebra 1.0.1 or later.

As of this release, Zebra requires Rust 1.70 to build. macOS builds are no longer officially supported by the Zebra team.

If you're running zebrad in a terminal, you'll see a new Zebra welcome message.

Please report bugs to the Zebra GitHub repository

Breaking Changes

This release has the following breaking changes:

• Zebra limits each IP address to 1 peer connection, to prevent denial of service attacks. This can be changed using the network.max_connections_per_ip config. (#6980, #6993, #7013).
  Thank you to @dimxy from komodo for reporting this bug, and the Ziggurat team for demonstrating
  its impact on testnet.
• Zebra uses new APIs in Rust 1.70 to prevent concurrency bugs that could cause hangs or panics
  (#7032).

Support Changes

These platforms are no longer supported by the Zebra team:

• macOS has been moved from tier 2 to tier 3 support (#6965). We disabled our regular macOS builds because Rust 1.70 causes crashes during shutdown on macOS x86_64 (#6812). Zebra's state uses database transactions, so it should not be corrupted by the crash.

Security

• Use Arc::into_inner() to avoid potential hangs or panics (#7032)
• Replace openssl with rustls in tests and experimental features (#7047)

Network Security

• Fix long delays in accepting inbound handshakes, and delays in async operations throughout Zebra. (#7103). Thank you to the Ziggurat Team for reporting this bug.
• Limit each IP address to 1 peer connection, to prevent denial of service attacks. (#6980, #6993)
• Close new peer connections from the same IP and port, rather than replacing the older connection (#6980)
• Reduce inbound service overloads and add a timeout (#6950)
• Stop panicking when handling inbound connection handshakes (#6984)

Added

• Make the maximum number of connections per IP configurable (#7013)
• Make it easier to modify Zebra's config inside the Docker image (#7045)
• Print a Zebra logo and welcome text if stderr is terminal (#6945, #7075, #7095, #7102)

Changed

• Move macOS to tier 3 support (#6965)
• Install from crates.io in the README, rather than a git release tag (#6977)
• Add extra timeout logging to peer TCP connections (#6969)

Fixed

• Stop overwriting custom user configs inside Zebra's Docker image (#7045)
• Stop Zebra using 100% CPU even when idle (#7103), thank you to james_katz for reporting this bug
• Avoid potential hangs in the tokio async runtime (#7094)
• Replace or add RPC content type header to support zcashd RPC examples (#6885)
• Make zebra-network licensing clearer (#6995)

Configuration

• Ignore error from loading config if running the 'generate' or 'download' commands (#7014)
• Apply force_color to panic logs (#6997)

Logging & Error Handling

• Log a zebra-network task cancel on shutdown, rather than panicking (#7078)
• Fix incorrect function spans in some logs (#6923, #6995)
• Replace a state validation chain length assertion with a NotReadyToBeCommitted error (#7072)

Experimental Feature Fixes

• Add an elasticsearch feature to block serialize to fix experimental build failures (#6709)
• Prevent progress bar from panicking by disabling limits that are never reached (#6940)

Contributors

Thank you to everyone who contributed to this release, we couldn't make Zebra without you:
@arya2, @conradoplg, @dconnolly, @dimxy from komodo, james_katz, @oxarbitrage, @teor2345, @upbqdn, and the Ziggurat team.

✨ Zebra 1.0.0 ✨

Zebra 1.0.0 - 2023-06-14

This is our 1.0.0 stable release.

This release also fixes a panic at startup when parsing the app version, publishes zebrad to crates.io, and publishes to Docker Hub under the latest tag.

Please report bugs to the Zebra GitHub repository

Security

• Avoid potential concurrency bugs in outbound handshakes (#6869)

Changed

• Publish to crates.io (#6908)
• Rename tower-batch to tower-batch-control (#6907)
• Upgrade to ed25519-zebra 4.0.0 (#6881)

Fixed

• Stop panicking at startup when parsing the app version (#6888)
• Avoid a race condition in testing modified JoinSplits (#6921)

Contributors

Thank you to everyone who contributed to this release, we couldn't make Zebra without you:
@dconnolly, @gustavovalverde, @oxarbitrage, @teor2345 and @upbqdn

Zebra 1.0.0-rc.9

Zebra 1.0.0-rc.9 - 2023-06-07

This release continues to address audit findings. It fixes multiple network protocol and RPC bugs,
and reduces sensitive information logging.

This is the last release candidate before the 1.0.0 stable release. Please report bugs to the Zebra GitHub repository

Breaking Changes

• The version subcommand has been replaced with a --version/-V flag (#6801)

Security

• Stop logging peer IP addresses, to protect user privacy (#6662)
• Stop logging potentially sensitive user information from unmined transactions (#6616)
• Rate-limit MetaAddrChange::Responded from peers (#6738)
• Ignore out of order Address Book changes, unless they are concurrent (#6717)
• Limit blocks and transactions sent in response to a single request (#6679)
• Rate-limit and size-limit peer transaction ID messages (#6625)
• Stop panicking on state RPC or block requests with very large heights (#6699)
• Try harder to drop connections when they shut down, Credit: Ziggurat Team (#6832)
• Randomly drop connections when inbound service is overloaded (#6790)

Added

• Report compiler version and Zebra features when starting Zebra (#6606)
• Update Zebra book summary to include supported platforms, platform tier policy, and versioning (#6683)
• Improve zebrad's help output, credit to @Rqnsom (#6801)
• Cache a list of useful peers on disk (#6739)
• Make the first stable release forward-compatible with planned state changes (#6813)

Fixed

• Limit RPC failure log length, add details to RPC failure logs (#6754)
• Allow inbound connections to Zebra running in Docker (#6755)
• Zebra now accepts filters for the start command when no subcommand is provided (#6801)
• Avoid panicking on state errors during shutdown (#6828)

Contributors

Thank you to everyone who contributed to this release, we couldn't make Zebra without you:
@arya2, @mpguerra, @oxarbitrage, @teor2345 and @upbqdn

Zebra 1.0.0-rc.8

Zebra 1.0.0-rc.8 - 2023-05-10

Starting in this release, Zebra has implemented an "end of support" halt. Just like zcashd, the zebrad binary will stop running 16 weeks after the last release date.
Also, this release adds the ZIP-317 rules to mempool transactions which should help with the Zcash network spam issue.

Security

• Avoid inbound service overloads and fix failing tests (#6537)
• Avoid a rare panic when a connection is dropped (#6566)
• Avoid some self-connection nonce removal attacks (#6410)
• Reject nodes using ZClassic ports, and warn if configured with those ports (#6567)

Added

• Add ZIP-317 rules to mempool (#6556)
• Add user agent argument to zebra-network crate (#6601)
• Refuse to run zebrad when release is too old (#6351)

Fixed

• Handle randomness generation and invalid random values as errors in cryptographic code (#6385)
• When configured for testnet, automatically use the correct testnet listener port (#6575)

Contributors

Thank you to everyone who contributed to this release, we couldn't make Zebra without you:
@arya2, @gustavovalverde, @oxarbitrage, @teor2345 and @upbqdn

Zebra 1.0.0-rc.7

This release features a security fix for unbounded memory use in zebra-network, introduces the "progress-bar" feature, and continues to address audit findings.

Security

• Limit the number of leftover nonces in the self-connection nonce set (#6534)
• Allow each initial peer to send one inbound request before disconnecting any peers (#6520)
• Limit the number of non-finalized chains tracked by Zebra (#6447)
• Update dependencies that only appear in the lock file (#6217)

Added

• Add confirmations to getrawtransaction method response (#6287)
• Add a config for writing logs to a file (#6449)
• Add an experimental terminal-based progress bar feature to Zebra, which is off by default (#6235)
• Create DockerHub image with mining enabled after each Zebra release (#6228)

Changed

• Increase ZIP-401 mempool cost thresholds for Orchard transactions (#6521)
• Suggest making sure the RPC endpoint is enabled for checkpointing (#6375)
• Refactor the handling of height differences (#6330)
• Upgrade shared dependencies to match zcashd 5.5.0 (#6536)
• Lookup unspent UTXOs in non-finalized state before checking disk (#6513)
• Stop re-downloading blocks that are in non-finalized side chains (#6335)

Fixed

• Validate header versions when serializing blocks (#6475)
• Stop ignoring new transactions after the mempool is newly activated (#6448)
• Fix off-by-one error in DNS seed peer retries, and clarify logs (#6460)
• Check that mempool transactions are valid for the state's chain info in getblocktemplate (#6416)
• Remove transactions with immature transparent coinbase spends from the mempool and block templates (#6510)
• Disable issue URLs for a known shutdown panic in abscissa (#6486)

Contributors

Thank you to everyone who contributed to this release, we couldn't make Zebra without you:
@arya2, @dconnolly, @gustavovalverde, @oxarbitrage, @teor2345 and @upbqdn

Zebra 1.0.0-rc.6

Zebra 1.0.0-rc.6 - 2023-03-23

In this release, we fixed several minor security issues, most notably hardening Zebra in response to the vulnerabilities recently disclosed by Halborn.

Known Issues

• orchard 0.3.0 can't verify halo2 proofs when compiled with Rust 1.69 or later (currently beta and nightly Rust). Compile Zebra with stable Rust to avoid this bug. (halo2/#737).
  Zebra tracking issue for this problem is #6232.

Security

• Harden Zebra's network protocol implementation in response to the Halborn disclosures (#6297)
• Bump incrementalmerkletree from 0.3.0 to 0.3.1, resolving a consensus bug on 32-bit platforms (#6258)
• Remove unused dependencies, and check for them in CI (#6216)
• Validate address length before reading (#6320, #6368)

Added

• Add instructions for mining with s-nomp to Zebra book (#6220)
• Export block data to elasticsearch database (#6274)
• Add elasticsearch section to Zebra book (#6295)

Changed

• Update Zebra's build instructions (#6273)
• Improve CommitBlockError message (#6251)

Fixed

• Stop ignoring some transaction broadcasts (#6230)

Contributors

Thank you to everyone who contributed to this release, we couldn't make Zebra without you:
@arya2, @dconnolly, @mpguerra, @oxarbitrage, @teor2345 and @upbqdn