Skip to content

Commit

Permalink
[storage-compression] Reorgnized
Browse files Browse the repository at this point in the history
  • Loading branch information
hyunsooda committed Feb 10, 2025
1 parent f5c0825 commit 1e97d19
Showing 1 changed file with 15 additions and 9 deletions.
24 changes: 15 additions & 9 deletions KIPs/kip-237.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,8 @@ This KIP proposes to compress the three data types to reduce the total disk size

### Compression
A node periodically compresses storage in a background thread and removes the original data once it is confirmed safe to do so.
Data is compressed in units called "chunks".
An optimal chunk size should balance a high compression rate with minimal decompression overhead.
Determining the ideal chunk capacity remains an open problem. However, our experiments suggest that a 1MB chunk size offers a favorable trade-off.

![compression-size](../assets/kip-237/chunk-selecting-compression.png)
![decompression](../assets/kip-237/chunk-selecting-decompression.png)
Data is compressed in units called "chunks". Selecting a good chunk size is described in

Currently, the chunk size is fixed at 1MB and cannot be configurable.

### Decompression
Decompression is required when a node receives a query (header, body, or receipts).
Expand All @@ -61,10 +55,22 @@ Evaluation on a [state-migrated mainnet (as of 2024-12-02)](https://packages.kai
| Body | 1100 | 267 | 75.7% |
| Receipts | 1600 | 179 | 88.8% |
| Others | ___ | ___ | - |
| Total | ___ | ___ | __._% |
| Total | ___ | ___ | 57.21% |

It took 3 days for background compression threads to complete.

### Ecosystem changes
## Rationale

### Appropriate chunk size
An optimal chunk size should balance a high compression rate with minimal decompression overhead.
Determining the ideal chunk capacity remains an open problem. However, our experiments suggest that a 1MB chunk size offers a favorable trade-off.

![compression-size](../assets/kip-237/chunk-selecting-compression.png)
![decompression](../assets/kip-237/chunk-selecting-decompression.png)

Currently, the chunk size is fixed at 1MB and cannot be configurable.

## Backwards Compatibility
An existing database can be compressed without node shutdown.
APIs querying the historic blocks, transactions, and receipts must work correctly because the data are not lost.

Expand Down

0 comments on commit 1e97d19

Please sign in to comment.