Skip to content

Commit

Permalink
Revise Database page
Browse files Browse the repository at this point in the history
  • Loading branch information
@rubo
rubo committed Nov 1, 2023
1 parent a49b22b commit 33835a4
Show file tree
Hide file tree
Showing 2 changed files with 204 additions and 81 deletions.
254 changes: 202 additions & 52 deletions docs/fundamentals/database.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,62 +3,212 @@ title: Database
sidebar_position: 5
---

Nethermind uses the [RocksDB](https://rocksdb.org) database to store state. By default the database is stored in the
same directory where the client binaries are. You can change it by providing a `--baseDbPath` config switch in the
command line, e.g.`./Nethermind.Runner --config goerli --baseDbPath /home/username/nethermind_db`

After Nethermind is started, you will see multiple directories appearing in the _baseDbPath_ directory.

![Example of the DB directory on a freshly deployed Ubuntu VM with Nethermind.](</img/image(61).png>)

| DB Directory | Content |
|-------------------|------------------------------------------------------------------------------------------------------------------------------------------|
| blockInfos | information about blocks at each level of the block tree (canonical chain and branches) |
| blocks | block bodies (block transactions and uncle data) |
| bloom | bloom indexes for fast log searches |
| canonicalHashTrie | LES protocol related data |
| code | contract bytecodes |
| discoveryNodes | peers discovered via discovery protocol - used for quick peering after restarts (you can copy this DB between nodes to speed up peering) |
| headers | block headers only |
| pendingTx | (this DB is wiped out on each restart) 2nd level cache of pending transactions / mempool (1st level is in memory) |
| peers | stores additional sync peers information (like peer reputation) - you can copy this DB between nodes to speed up peering on fresh sync |
| receipts | transaction receipts data |
| state | blockchain state including accounts and contract storage (Patricia trie nodes) |

You can use `rsync` between your nodes to clone the database (One of our users copied entire 4.5TB archive state this
way while the node was running and only stopped the node for the very last stage of `rsync` ). You can also simply copy
the database between Unix and Windows systems (and most likely macOS).

## How to reduce database size
import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";

Nethermind uses the [RocksDB](https://rocksdb.org) database to store the state. By default, the database is located in the
same directory where the Nethermind executable is. You can change the database location using the [`-d, --baseDbPath`](configuration.md#basic-options) command line option.

## Database directory structure

| Directory | Description |
| - | - |
| blockInfos | Information about blocks at each level of the block tree (canonical chain and branches) |
| blocks | Block bodies (block transactions and uncles) |
| bloom | Bloom indices for fast log searches |
| canonicalHashTrie | LES protocol related data |
| code | Contract bytecodes |
| discoveryNodes | Peers discovered via discovery protocol - used for quick peering after restarts (you can copy this DB between nodes to speed up peering) |
| headers | Block headers only |
| pendingTx | The second level cache of pending transactions/mempool (the first level is in memory). Wiped out on each restart. |
| peers | Additional sync peers information (like peer reputation) - you can copy this DB between nodes to speed up peering on fresh sync |
| receipts | Transaction receipts |
| state | Blockchain state including accounts and contract storage (Patricia trie nodes) |

You can use `rsync` between your nodes to clone the database (One of our users copied the entire 4.5TB archive state this
way while the node was running and only stopped the node for the very last stage of `rsync` ). You can also copy
the database between Linux, Windows, and macOS.

## Database size

The table below presents the Nethermind database size after fresh sync with different configurations and networks (as of
May 2023).

| Network/configuration | Database size after fresh sync |
|-----------------------------------------|--------------------------------|
| Mainnet (default) | 504 GB |
| Mainnet without ancient barriers | 664 GB |
| Mainnet archive | 14 TB |
| Mainnet without old bodies and receipts | 178 GB |
| Gnosis | 112 GB |
| Goerli | 240 GB |

### Database size in details

Below is a comprehensive list of the supported chains, along with a detailed breakdown of their respective database directories. For reference, the database sizes listed are based on the data from July 2023 and have been determined using the standard configurations provided.

<Tabs>
<TabItem value="mainnet" label="Mainnet">

| Database | Mainnet | Mainnet with ancient barriers |
|-----------|------------|-------------------------------|
| State | 161 GB | 161 GB |
| Receipts | 152 GB | 104 GB |
| Blocks | 334 GB | 222 GB |
| Bloom | 5.3 GB | 5.3 GB |
| Headers | 8.1 GB | 8.1 GB |
| Code | 3 GB | 3 GB |
| Other | ... | .. |
| **Total** | **664 GB** | **504 GB** |

</TabItem>
<TabItem value="goerli" label="Goerli">

| Directory | Size |
| ----------------- | ---------- |
| state | 51 GB |
| receipts | 35 GB |
| blocks | 185 GB |
| code | 7.4 GB |
| peers | 8 KB |
| witness | 4.1 MB |
| blockInfos | 417 MB |
| bloom | 2.5 GB |
| headers | 3.5 GB |
| canonicalHashTrie | 4.1 MB |
| metadata | 4.4 MB |
| discoveryNodes | 1.5 MB |
| **TOTAL** | **283 GB** |

</TabItem>
<TabItem value="sepolia" label="Sepolia">

| Directory | Size |
| ----------------- | --------- |
| state | 8.5 GB |
| receipts | 8.2 GB |
| blocks | 36 GB |
| code | 1.5 GB |
| peers | 8 KB |
| witness | 4.1 MB |
| blockInfos | 177 MB |
| bloom | 1.3 GB |
| headers | 1.2 GB |
| canonicalHashTrie | 4.1 MB |
| metadata | 4.1 MB |
| discoveryNodes | 1.7 MB |
| **TOTAL** | **56 GB** |

</TabItem>
<TabItem value="gnosis" label="Gnosis">

| Directory | Size |
| ----------------- | ------ |
| state | 40 GB |
| receipts | 21 GB |
| blocks | 43 GB |
| code | 350 MB |
| peers | 8 KB |
| witness | 4.1 MB |
| blockInfos | 1.5 GB |
| bloom | 7.5 GB |
| headers | 9.4 GB |
| canonicalHashTrie | 4.1 MB |
| metadata | 4.2 MB |
| discoveryNodes | 1.8 MB |
| **TOTAL** | **121 GB** |

</TabItem>
<TabItem value="chiado" label="Chiado">

| Directory | Size |
| ----------------- | ---------- |
| state | 1.6 GB |
| receipts | 646 MB |
| blocks | 2.3 GB |
| code | 22 MB |
| peers | 8 KB |
| witness | 4.1 MB |
| blockInfos | 224 MB |
| bloom | 1.4 GB |
| headers | 1.1 GB |
| canonicalHashTrie | 4.1 MB |
| metadata | 4.1 MB |
| discoveryNodes | 1.7 MB |
| **TOTAL** | **7.1 GB** |

</TabItem>
<TabItem value="energyweb" label="Energy Web">

| Directory | Size |
| ----------------- | ------------ |
| state | 31.13 GB |
| receipts | 5.13 GB |
| blocks | 15.55 GB |
| code | 7.20 MB |
| peers | 311 B |
| witness | 103.11 KB |
| blockInfos | 1.22 GB |
| bloom | 7.78 GB |
| headers | 6.70 GB |
| canonicalHashTrie | 103.35 KB |
| metadata | 103.31 KB |
| discoveryNodes | 1.68 MB |
| **TOTAL** | **67.52 GB** |

</TabItem>
<TabItem value="volta" label="Volta">

| Directory | Size |
| ----------------- | ------------ |
| state | 35.71 GB |
| receipts | 7.67 GB |
| blocks | 25.11 GB |
| code | 75.14 MB |
| peers | 616 B |
| witness | 119.95 KB |
| blockInfos | 1.15 GB |
| bloom | 7.40 GB |
| headers | 6.94 GB |
| canonicalHashTrie | 120.00 KB |
| metadata | 119.97 KB |
| discoveryNodes | 1.55 MB |
| **TOTAL** | **84.06 GB** |

</TabItem>
</Tabs>

## Reducing database size

The Nethermind database can experience substantial growth over time, starting from an initial size of approximately 650
GB. As a result, many node-setups are configured to run on 1TB disks. However, even with the application of settings
GB. As a result, many node setups are configured to run on 1 TB disks. However, even with settings
designed to slow the growth rate, these disks may eventually run out of free space.

Current options to reduce db size are:
The current options to reduce the database size are as follows:

1. [Re-sync database from scratch](sync.md#resync-a-node-from-scratch)
2. [Full pruning](pruning.md)
- [Resyncing database from scratch](sync.md#resync-a-node-from-scratch)
- [Pruning](pruning.md)

The table below presents a short comparison of these methods including possible fine-tuning of each method. Data was
fetched from a node running on a machine with the below specifications:\
\
**Node Version**: 1.18.0\
**Consensus Client**: Lighthouse\
**CPU**: AMD EPYC 7713 64-Core Processor (16 cores allocated for this VM)\
**RAM**: 64GB\
**Disk Size**: 1.2 TB\
**IOPS of the disk\***: 70,000 - 80,000 IOPS (results vary in this range)

| Metric | Resync | Full Pruning | Full Pruning + Memory Budget (4 GB) |
|----------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|-------------------------------------------------------------|
| **Time of execution** | ~4h | ~24h | ~12h |
| **Minimum free disk space** | N/A - you can execute resync even if there is 0 free space (avoid such case) | 250 GB | 250 GB |
| **Attestation rate drop** | 100% - no attestation rewards during that time or highly reduced | 5-10% during that time. | TBC - not enough data yet |
| **Average block processing time of new blocks during process** | N/A - new blocks are being processed after state but are significantly slower until OldBodies and OldReceipts are downloaded. Afterwards, average about 0.35s | 0.7s | 1.0s |
| **Is node online during process?** | No - unless State is synced | Yes - node follows chain and all modules are still enabled. | Yes - node follows chain and all modules are still enabled. |

:::info
Command used for testing IOPS of disk:\
`fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw`
:::
fetched from a node running on a machine with the below specifications:

- **Node.js**: v1.18.0
- **Consensus client**: Lighthouse
- **CPU**: AMD EPYC 7713 (16 cores allocated for the VM)
- **RAM**: 64 GB
- **Disk size**: 1.2 TB
- **Disk IOPS**: 70,000 to 80,000

| Metric | Resync | Pruning | Pruning and memory budget (4 GB) |
| - | - | - | - |
| **Execution time** | ~4h | ~24h | ~12h |
| **Minimum free disk space** | N/A. You can execute resync even if there is 0 free space (avoid such a case). | 250 GB | 250 GB |
| **Attestation rate drop** | 100%. No attestation rewards during that time or highly reduced. | 5–10% during that time | N/A |
| **Average block processing time of new blocks during the process** | N/A. New blocks are processed after state but are significantly slower until old bodies/receipts are downloaded. Afterward, average about 0.35s. | 0.7s | 1.0s |
| **Is the node online during the process?** | No, unless the state is synced. | Yes. The node follows the chain, and all modules are still enabled. | Yes. The node follows chain and all modules are still enabled. |

The command used for testing disk IOPS was as follows:

```bash
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw
```
31 changes: 2 additions & 29 deletions docs/get-started/system-requirements.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ Nonetheless, for a comprehensive understanding of disk growth and usage, we reco
Nethermind requires approximately 504 GB of disk space (as of May 2023) after a fresh Ethereum Mainnet sync using default parameters. This size increases over time as the Ethereum chain grows. The node's database is in its most optimal state immediately after a sync or full pruning. Following the initial sync, the database grows at a rate of approximately 27 GB per week. To maintain this process, occasional resyncing or pruning of the node is necessary to bring it back to its optimal database size. For more info on managing node disk usage growth, see [how to reduce database size](../fundamentals/sync.md#resync-a-node-from-scratch).<br />
Having a larger disk space allocation reduces the need for frequent maintenance and alleviates concerns about the chain outgrowing the available space. It's worth noting that the only drawback of running a smaller disk is the requirement for more regular resyncing or pruning. We believe that a 2 TB disk will suffice for most users. However, the choice between 1 TB and 2 TB depends on factors such as hardware costs, cloud provider expenses, and individual requirements.

For more details, see [Database size](../fundamentals/database.md#database-size).

#### Disk speed

The speed of the disk often acts as a bottleneck for the node's performance. It is crucial for optimizing your validator's performance rewards and the syncing process. We highly recommend a disk with a minimum of 10,000 IOPS for both write and read operations. Slower disks may hinder your ability to synchronize the blockchain successfully.
Expand All @@ -65,32 +67,3 @@ A node can be run using different syncing options, and each option has different
- **Non-validator mode.** This mode drops all historical receipts and bodies, but it cannot be used for validation.

For more details, see [sync modes](../fundamentals/sync.md).

#### Database size with different configurations

The below table presents the Nethermind database size after fresh sync with different configurations and networks (as of
May 2023).

| Network/configuration | Database size after fresh sync |
|-----------------------------------------|--------------------------------|
| Mainnet (default) | 504 GB |
| Mainnet without ancient barriers | 664 GB |
| Mainnet archive | 14 TB |
| Mainnet without old bodies and receipts | 178 GB |
| Gnosis | 112 GB |
| Goerli | 240 GB |

#### Database in details

A freshly fast-synced Mainnet disk space with details about specific databases (as of May 2023).

| Database | Mainnet | Mainnet with ancient barriers |
|-----------|------------|-------------------------------|
| State | 161 GB | 161 GB |
| Receipts | 152 GB | 104 GB |
| Blocks | 334 GB | 222 GB |
| Bloom | 5.3 GB | 5.3 GB |
| Headers | 8.1 GB | 8.1 GB |
| Code | 3 GB | 3 GB |
| Other | ... | .. |
| **Total** | **664 GB** | **504 GB** |

0 comments on commit 33835a4

Please sign in to comment.