-
Notifications
You must be signed in to change notification settings - Fork 78
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
services: add new service for fetching blocks from NeoFS
Close #3496 Co-authored-by: Anna Shaleva <[email protected]> Signed-off-by: Ekaterina Pavlova <[email protected]>
- Loading branch information
Showing
2 changed files
with
92 additions
and
79 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,106 +1,74 @@ | ||
| # NeoFS BlockStorage | ||
| This service was proposed as a part of [ Using NeoFS to store blocks and snapshots #3463](https://github.com/neo-project/neo/issues/3463). | ||
| It is an alternative to the P2P synchronization protocol by using NeoFS to | ||
| store and retrieve blockchain blocks and snapshots. This improves | ||
| synchronization efficiency, reduces local storage, and streamlines node | ||
| operations. | ||
| # NeoFS block storage | ||
|
|
||
| Using NeoFS to store chain's blocks and snapshots was proposed in | ||
| [#3463](https://github.com/neo-project/neo/issues/3463). NeoGo contains several | ||
| extensions utilizing NeoFS block storage aimed to improve node synchronization | ||
| efficiency and reduce node storage size. | ||
|
|
||
| ## Components and functionality | ||
|
|
||
| ### Storage schema | ||
| A single container is used to store blocks and index files. Each block | ||
| is stored as a separate object with a unique OID and a set of attributes: | ||
| - block index | ||
| - block primary index | ||
| - size. | ||
| ### Block storage schema | ||
|
|
||
| Index files contain OIDs of ordered blocks. Each index file is an object | ||
| containing a constant number of OIDs in | ||
| binary form, where every number represent the order of the index file. | ||
| A single NeoFS container is used to store blocks and index files. Each block | ||
| is stored in a binary form as a separate object with a unique OID and a set of | ||
| attributes: | ||
| - block object identifier with block index value (`block:1`) | ||
| - primary node index (`primary:0`) | ||
| - block hash in the LE form (`hash:5412a781caf278c0736556c0e544c7cfdbb6e3c62ae221ef53646be89364566b`) | ||
| - previous block hash in the LE form (`prevHash:3654a054d82a8178c7dfacecc2c57282e23468a42ee407f14506368afe22d929`) | ||
| - millisecond-precision block timestamp (`time:1627894840919`) | ||
|
|
||
| Attributes have the following structure: `AttributeName:IntValue`. | ||
| In the container are stored blocks with attributes `block:1`, | ||
| `block:2`, etc. And index files with attributes `oid:1`, `oid:2`, etc. | ||
| Each index file is an object containing a constant-sized batch of raw block object | ||
| IDs in binary form ordered by block index. Each index file is marked with the | ||
| following attributes: | ||
| - index file identifier with consecutive file index value (`index:0`) | ||
| - the number of OIDs included into index file (`size:128000`) | ||
|
|
||
| ### NeoFS BlockFetcher | ||
| The NeoFS BlockFetcher service is designed as an alternative to P2P | ||
| synchronisation protocol. It allows to fetch blocks, primarily used at | ||
| the start of a node's lifecycle, providing an alternative to downloading | ||
| blocks through the P2P NeoGo network. By default, the service is disabled. | ||
|
|
||
| The NeoFS BlockFetcher service has two modes of operation: | ||
| - Index File Search: Search for index files, which contain OIDs of blocks, | ||
| and fetch blocks from NeoFS by provided OIDs. | ||
| - Direct Block Search: Search and fetch blocks directly from NeoFS container. | ||
| NeoFS BlockFetcher service is designed as an alternative to P2P synchronisation | ||
| protocol. It allows to download blocks from a trusted container in the NeoFS network | ||
| and persist them to database using standard verification flow. NeoFS BlockFetcher | ||
| service primarily used during the node's bootstrap, providing a fast alternative to | ||
| P2P blocks synchronisation. | ||
|
|
||
| NeoFS BlockFetcher service has two modes of operation: | ||
| - Index File Search: Search for index files, which contain batches of block object | ||
| IDs and fetch blocks from NeoFS by retrieved OIDs. | ||
| - Direct Block Search: Search and fetch blocks directly from NeoFS container via | ||
| built-in NeoFS object search mechanism. | ||
|
|
||
| Operation mode of BlockFetcher can be configured via `SkipIndexFilesSearch` | ||
| parameter. | ||
|
|
||
| #### Operation flow | ||
|
|
||
| ### BlockFetcher operation flow | ||
| 1. **OID Fetching**: | ||
| Depending on the mode, the service either: | ||
| - Searches for index files by attribute and reads blocks OIDs from it. | ||
| - Searches batches of blocks directly by attributes. | ||
| - Searches for index files by index file attribute and reads block OIDs from index | ||
| file object-by-object. | ||
| - Searches batches of blocks directly by block attribute (the batch size is | ||
| configured via `OIDBatchSize` parameter). | ||
|
|
||
| Once the OIDs are retrieved, they are immediately redirected to the | ||
| block downloading routines for further processing. The channel that | ||
| is used to redirect block OIDs to downloading routines is buffered | ||
| to provide smooth OIDs delivery without delays. Size of this channel | ||
| can be configured via `OIDBatchSize` parameter. | ||
| to provide smooth OIDs delivery without delays. The size of this channel | ||
| can be configured via `OIDBatchSize` parameter and equals to `2*OIDBatchSize`. | ||
| 2. **Parallel Block Downloading**: | ||
| The number of downloading routines can be configured via | ||
| `DownloaderWorkersCount` parameter. It's up to the user to find the | ||
| balance between the downloading speed and blocks persist speed for every | ||
| node that uses NeoFS BlockFetcher. Downloaded blocks are placed into a | ||
| buffer with a size of `BQueueSize` parameter for further processing. | ||
| buffered channel of size `IDBatchSize` with further redirection to the | ||
| block queue. | ||
| 3. **Block Insertion**: | ||
| Downloaded blocks are inserted into the blockchain using the same logic | ||
| as in the P2P synchronisation protocol. The bqueue is used to store | ||
| as in the P2P synchronisation protocol. The block queue is used to order | ||
| downloaded blocks before they are inserted into the blockchain. The | ||
| size of the bqueue can be configured via the `BQueueSize` parameter | ||
| size of the queue can be configured via the `BQueueSize` parameter | ||
| and should be larger than the `OIDBatchSize` parameter to avoid blocking | ||
| the downloading routines. | ||
|
|
||
| Once all blocks available in the NeoFS container are processed, the service | ||
| shuts down automatically. | ||
|
|
||
| ### NeoFS BlockFetcher Configuration | ||
| `NeoFSBlockFetcher` configuration section contains settings for NeoFS | ||
| BlockFetcher module and has the following structure: | ||
| ``` | ||
| NeoFSBlockFetcher: | ||
| Enabled: true | ||
| UnlockWallet: | ||
| Path: "./wallet.json" | ||
| Password: "pass" | ||
| Addresses: | ||
| - st1.storage.fs.neo.org:8080 | ||
| Timeout: 10m | ||
| DownloaderWorkersCount: 500 | ||
| OIDBatchSize: 8000 | ||
| BQueueSize: 16000 | ||
| SkipIndexFilesSearch: false | ||
| ContainerID: "EPGuD26wYgQJbmDdVBoYoNZiMKHwFMJT3A5WqPjdUHxH" | ||
| BlockAttribute: "block" | ||
| IndexFileAttribute: "oid" | ||
| ``` | ||
| where: | ||
| - `Enabled` enables NeoFS BlockFetcher module. | ||
| - `UnlockWallet` contains wallet settings to retrieve account to sign requests to | ||
| NeoFS. Without this setting, the module will use randomly generated private key. | ||
| For configuration details see [Unlock Wallet Configuration](https://github.com/nspcc-dev/neo-go/blob/master/docs/node-configuration.md#unlock-wallet-configuration) | ||
| - `Addresses` is a list of NeoFS storage nodes addresses. | ||
| - `Timeout` is a timeout for a single request to NeoFS storage node. | ||
| - `ContainerID` is a container ID to fetch blocks from. | ||
| - `BlockAttribute` is an attribute name of NeoFS object that contains block | ||
| data. | ||
| - `IndexFileAttribute` is an attribute name of NeoFS object that contains OIDs of | ||
| blocks objects. | ||
| - `DownloaderWorkersCount` is a number of workers that download blocks from | ||
| NeoFS. | ||
| - `OIDBatchSize` is the number of blocks to search per a single request to NeoFS | ||
| in case of disabled index files search. Also, for both modes of BlockFetcher | ||
| operation this setting manages the buffer size of OIDs and blocks transferring channels. | ||
| - `BQueueSize` is a size of the block queue used to manage consecutive blocks | ||
| addition to the chain. It must be larger than | ||
| `OIDBatchSize` and highly recommended to be 2*`OIDBatchSize` or 3*`OIDBatchSize`. | ||
| - `SkipIndexFilesSearch` is a flag that allows skipping index files search | ||
| in NeoFS storage nodes and search for blocks directly. It is set to `false` | ||
| by default. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters