Skip to content

Commit

Permalink
Basic transparent partial-key badger implementation
Browse files Browse the repository at this point in the history
By a wide margin the top issue with large blockstores is the size of their index.

Transparently switch the badger lookup key to the leading 128 bits of the hash,
regardless of type/codec. In the proccess forbid AllKeysChan and rewrite ForEachKey.

Also unconditionally enable HashOnRead (only safe way to do partial keys)

This is safe (and has been tested) to run on an existing lotus install, with splitstore
enabled and compacting properly. When converting an existing setup simply set the
envvar LOTUS_CHAIN_BADGERSTORE_QUERY_LEGACY_KEYS to true. When sufficient amount of
FullGC cycles have passed and all the keys in your blockstore are "new" - the envvar
is no longer needed.

Reasoning:

The filecoin blockstore in April 2024 sits at about 26 billion blocks. The
current scheme of /blocks/base32-enc-of-entire-mh puts a key at 66 bytes, for
a total raw index of about 1.6TiB.

Reduction of every key to mere 16 bytes reduces this to just 390GiB, potentially
even fitting in memory.
  • Loading branch information
ribasushi committed May 16, 2024
1 parent 2337c50 commit 043787c
Show file tree
Hide file tree
Showing 7 changed files with 339 additions and 268 deletions.
Loading

0 comments on commit 043787c

Please sign in to comment.