[skip changelog] 11987 part 2 #12403
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…23 (filecoin-project#12276) * Bump filecoin-ffi to v1.28.0 Bump filecoin-ffi to v1.28.0
* Test various multi-sector cases with niporep * UnmanagedMiner can now onboard multiple sectors for one miner and manage PoSt for all of them
… test (filecoin-project#12203) * fix the event bloom filter in eth tx receipts
…2332) * fix bug in receipt for txtype for legacy txns * update ChangeLog * Update CHANGELOG.md Co-authored-by: Rod Vagg <[email protected]> --------- Co-authored-by: Rod Vagg <[email protected]>
* proofparams alternate * createminer * const factored from /build and types updated to use it * buildconstants for more places * deprecate msg * itest cleanup * alerting interface * house cleaning * rm policy and drand from buildconstants * clean up curio further * aussie waffle * pr fixes * fix lints * little fixes * oops this got updated * unbreak test builds * test fixes * comments - cleanups * itests fix alerting * rm obsolete alertinginterface * spelling oops * changelog * tests need buildconstants port * Fully migrate BlockGasTarget * ulimit should not depend on build * complete the simplest deprecations * bringing back versions
* chore: s/CidMustParse/cid.MustParse/g * chore: fix sloppy init duplication * chore: move the policy init() to the core namespace * chore: remove unused proxies for policy.DealDurationBounds
While technically the entire "what are we building" should stay in build/... , not having the static-ish machinery split across 2 namespaces is an improvement
This is a large diff, yet should have exactly zero functional changes Ideally as a result of this some parts of the depchain will become lighter, with downstream reaping the same benefits as the team that initiated this split. P.S. work was done while forming better intuition of current dependency graph
Makes testground build again, namely: for d in ./cmd/lotus*; do go build -tags testground -o /dev/null $d || break ; done
Co-authored-by: AnomalRoil <[email protected]>
Co-authored-by: Rod Vagg <[email protected]>
…ilecoin-project#12324) - `` Co-authored-by: Rod Vagg <[email protected]> --------- Co-authored-by: Rod Vagg <[email protected]>
Signed-off-by: riskrose <[email protected]>
Minor API changes: * gateway.NewRateLimiterHandler and gateway.NewConnectionRateLimiterHandler have been replaced with gateway.NewRateLimitHandler. * The handlers returned by both gateway.NewRateLimitHandler and the primary gateway.Handler return an http.Handler augmented with a Shutdown(ctx) method to be used for graceful cleanup of resources. Fix: * --per-conn-rate-limit was previously applied as a global rate limiter, effectively making it have the same impact as --rate-limit. This change fixes the behaviour such that --per-conn-rate-limit is applied as a API call limiter within a single connection (i.e. a WebSocket connection). The rate is specified as tokens-per-second, where tokens are relative to the expense of the API call being made.
* CLI takes --eth-max-filters-per-conn * gateway.NewNode() now takes the API plus an optional list of Options
Co-authored-by: Phi-rjan <[email protected]>
Update Building_a_network_skeleton.md with the steps needed for adding support for nvXX in lotus-shed cmd´s
* chore: docs: Update dependencies in Lotus chore: docs: Update dependencies in Lotus * Update documentation/misc/Building_a_network_skeleton.md Co-authored-by: Rod Vagg <[email protected]> --------- Co-authored-by: Rod Vagg <[email protected]>
By a wide margin the top issue with large blockstores is the size of their index. Transparently switch the badger lookup key to the leading 128 bits of the hash, regardless of type/codec. In the proccess forbid AllKeysChan and rewrite ForEachKey. Also unconditionally enable HashOnRead (only safe way to do partial keys) This is safe (and has been tested) to run on an existing lotus install, with splitstore enabled and compacting properly. When converting an existing setup simply set the envvar LOTUS_CHAIN_BADGERSTORE_QUERY_LEGACY_KEYS to true. When sufficient amount of FullGC cycles have passed and all the keys in your blockstore are "new" - the envvar is no longer needed. Reasoning: The filecoin blockstore in April 2024 sits at about 26 billion blocks. The current scheme of /blocks/base32-enc-of-entire-mh puts a key at 66 bytes, for a total raw index of about 1.6TiB. Reduction of every key to mere 16 bytes reduces this to just 390GiB, potentially even fitting in memory.
Prefix every bin-key value with a single uint64 varint containing:
(3bit store-type) // 0 for as-is, 1 for basic zstd compression, remainder reserved
+
(one reserved bit) << 3 // always 0 for now
+
(3bit compressability) << 4 // 0 if store-type is 0, otherwise `c := (origLen-compLen) * 8 / origLen`
// (using integer math, `origLen > compLen > 0` holds for any non-0 store-type)
+
(IPLD block data size) << 7 // 0 if store-type is 0
Include a rudimentary, dictionary-less zstd compressor as the first
non-verbatim storage type 1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues
#11987 (comment)