Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce Peapod #2462

Merged
merged 6 commits into from
Aug 15, 2023
Merged

Introduce Peapod #2462

merged 6 commits into from
Aug 15, 2023

Commits on Aug 15, 2023

  1. Introduce new storage component Peapod 

    Currently, storage node saves relatively small NeoFS objects in
Blobovnicza tree component: group of BoltDB wrappers managed as a tree.
This component has pretty complex data structure, code implementation
and dubious performance results.

Peapod is a new storage component introduced to replace Blobovnicza one
as more simple and effective. It also bases on single BoltDB instance,
but organizes batch writes in a specific way. In future, Peapod is going
to be used as a storage of small objects by the BlobStor.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
    @cthulhu-rider
    cthulhu-rider committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    b28c1f2 View commit details
    Browse the repository at this point in the history

  2. node/config: Fix type in FSTree config docs 

    Signed-off-by: Leonard Lyubich <[email protected]>
    @cthulhu-rider
    cthulhu-rider committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    ec80229 View commit details
    Browse the repository at this point in the history

  3. Introduce tool to migrate objects from Blobovnicza tree to Peapod 

    Add `cmd/blobovnicza-to-peapod` application which accepts YAML
configuration file of the storage node and, for each configured shard,
overtakes data from Blobovnicza tree to Peapod created in the parent
directory.

The tool is going to be used for phased and safe rejection of the
Blobovnicza trees and the transition to Peapods.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
    @cthulhu-rider
    cthulhu-rider committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    e36ad52 View commit details
    Browse the repository at this point in the history

  4. blobstortest: Use errors.Is to asset iteration error 

    Error wrapping is normal, so we should always be ready to it.

Signed-off-by: Leonard Lyubich <[email protected]>
    @cthulhu-rider
    cthulhu-rider committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    c5112c4 View commit details
    Browse the repository at this point in the history

  5. node: Support Peapod as BlobStor's sub-storage 

    Support `peapod` sub-storage type in BlobStor configuration.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
    @cthulhu-rider
    cthulhu-rider committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    c060b16 View commit details
    Browse the repository at this point in the history

  6. node: Make Peapod's flush interval configurable 

    There may be a need to tune time interval b/w batch writes to disk in
Peapod component.

Add storage node's config with `flush_interval` key of type duration
defaulting to 10ms.

Signed-off-by: Leonard Lyubich <[email protected]>
    @cthulhu-rider
    cthulhu-rider committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    b01baf6 View commit details
    Browse the repository at this point in the history