tapdb: switch proof cache to size-based limits and add deep size estimation #1870
Conversation
Refactor the function by introducing an internal helper to improve readability.
Refactor MacaroonWhitelist to handle public universe proof courier permissions independently. This clarifies the logic and avoids coupling courier access with other universe server permissions.
Remove the defaultMacaroonWhitelist map and inline its entries directly into the conditional logic within MacaroonWhitelist. This ensures that access to previously always-available endpoints is now governed by explicit user configuration (read/write/courier), improving permission control and aligning with expected access restrictions.
Separate the mint anchor transaction fee rate calculation from fundGenesisPsbt into anchorTxFeeRate. This refactor is part of a broader effort to simplify calling fundGenesisPsbt from unit tests.
Extract the wallet funding call into a closure that is passed as an argument. This prepares fundGenesisPsbt to become a standalone function, making it easier to call in unit tests.
Pass the pending batch and chain params into fundGenesisPsbt and convert it into a standalone function rather than a method on ChainPlanter. This change makes it easier to call fundGenesisPsbt from unit tests.
The batch key was only used for logging. This commit moves the log messages outside fundGenesisPsbt, simplifying the function for better code health.
The mock helper FundGenesisTx now returns the index of the change output. It also dynamically computes the index of the change output it adds. These enhancements will be useful when handling packets with supply pre-commitment outputs.
The funding routine now uses the refactored fundGenesisPsbt function, introduced in a previous commit. Which adds test coverage for the batch funding logic. An optional argument is also added to allow skipping funding.
RandSeedlingMintingBatch retires.
Refactored GetBlockTimestamp to call GetBlockHeaderByHeight and return an optional error type. Removed the timestamp-to-block-height cache, as it did not handle re-orgs correctly. This prepares the codebase for a more comprehensive caching mechanism to be added in a follow-up commit.
Introduce a reusable cache that stores full headers keyed by height and hash. Tracks confirmation depth and treats shallow entries as unsettled (return miss). Detects conflicts at a height and invalidates shallower headers on reorg. Size and random purge fraction are configurable (default 100k entries, 10 percent). Not yet used by LndRpcChainBridge.
…whitelist Improve `MacaroonWhitelist` Structure and Permission Granularity
Adds a block header cache to LndRpcChainBridge, which indirectly improves performance of methods like GetBlockTimestamp by avoiding repeated block header fetches.
Refactor `fundGenesisPsbt` and enhance test helpers for coverage
lndservices: add reorg aware block header cache; use in ChainBridge
Allow the creation of multiple vPackets when burning by group key.
Return multiple proofs in BurnAssetResponse.
…-branches ci: run CI on *-staging branches in addition to main
Summary of ChangesHello @ffranr, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly refactors the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a significant improvement by switching the proof cache from count-based to size-based limits, incorporating deep size estimation for more accurate memory management. The changes also include finer-grained eviction strategies for cache entries and enhanced error handling and logging around cache initialization. The new LowerBoundByteSize function, utilizing reflect and unsafe, provides a conservative estimate of an object's memory footprint, which is crucial for the new cache mechanism. Comprehensive unit tests have been added for this new functionality, covering various data types and edge cases like pointer cycles. Configuration options have been updated to support human-readable size values for the cache limit. Overall, the changes align well with the stated objective of improving cache efficiency and control.
Introduce `LowerBoundByteSize` for deep-size estimation of various data types, handling primitives, structs, slices, maps, and pointer cycles. Add unit tests to ensure correctness and coverage.
ffranr
force-pushed
the
wip/limit-proof-cache-size
branch
from
f273430 to
e2c0aad
Compare
- Replace `ProofKey` with `UniverseProofKey` for clearer cache key distinction, enabling removal of individual proofs by leaf key instead of all proofs for a universe ID. - Refactor proof cache methods: add `RemoveLeafKeyProofs` and enhance `RemoveUniverseProofs` for more granular control. - Implement deep size estimation in the `Size` method to improve memory management accuracy. - Change cache size limit from number of proofs to total memory size of all proofs, as proof sizes can vary due to inclusion of parent proof files.
- Replace `proofs-per-universe` configuration with `max-proof-cache-size`, allowing proof cache limits to be defined by total memory size instead of proof count. - Add logic to parse and handle human-readable size values (e.g., "32MB"). - Update default configuration and related comments. - Adjust `NewMultiverseStore` to use parsed cache size values and improve error handling.
ffranr
force-pushed
the
wip/limit-proof-cache-size
branch
from
e2c0aad to
7e26748
Compare
Pull Request Test Coverage Report for Build 19273545147Details
💛 - Coveralls |
Also, enhance the release notes template with a section to aid in reporting breaking config changes.
Updates the dependency to expose the Size() method on cache implementations, enabling direct access to cache size.
* Add support for an optional cache size callback to newCacheLogger. * Enable cache size logging in the universe proof cache.
darioAnongba
force-pushed
the
0-8-0-staging
branch
from
2b3ac4f to
035a840
Compare
|
@ffranr, remember to re-request review from reviewers when ready |
Issue: #1865
This PR introduces a size-based proof cache and supporting deep-size logic.
Summary
LowerBoundByteSize) and apply it to proof cache entries.Note
It's pretty trivial to get from here to logging cache size, but we need lightninglabs/neutrino#330 for that.