Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Basic transparent partial-key badger implementation
By a wide margin the top issue with large blockstores is the size of their index. Transparently switch the badger lookup key to the leading 128 bits of the hash, regardless of type/codec. In the proccess forbid AllKeysChan and rewrite ForEachKey. Also unconditionally enable HashOnRead (only safe way to do partial keys) This is safe (and has been tested) to run on an existing lotus install, with splitstore enabled and compacting properly. When converting an existing setup simply set the envvar LOTUS_CHAIN_BADGERSTORE_QUERY_LEGACY_KEYS to true. When sufficient amount of FullGC cycles have passed and all the keys in your blockstore are "new" - the envvar is no longer needed. Reasoning: The filecoin blockstore in April 2024 sits at about 26 billion blocks. The current scheme of /blocks/base32-enc-of-entire-mh puts a key at 66 bytes, for a total raw index of about 1.6TiB. Reduction of every key to mere 16 bytes reduces this to just 390GiB, potentially even fitting in memory.
- Loading branch information