Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

caplin: non-archive node prune epoch length option #12042

Open
errge opened this issue Sep 19, 2024 · 0 comments
Open

caplin: non-archive node prune epoch length option #12042

errge opened this issue Sep 19, 2024 · 0 comments
Labels

Comments

@errge
Copy link
Contributor

errge commented Sep 19, 2024

Rationale

When requesting data from the beacon chain using caplin RPC, currently there are two possibilities: the finalized epoch and the previous finalized epoch (and anything newer) is always reachable, meaning that the slots belonging to them can be used in RPC requests. Anything older returns 404, by default.

The second possibility, is to enable archival for caplin, which downloads all historical blocks and then processes them from the beginning towards the current time, and this processing (reported as "State processing progress") takes ~4-5 days on 3.0.0-alpha3 with modern hardware and full nvme setup with 64GB ram. During this processing, only the less interesting part of the chain (the beginning) is reachable, and later blocks become available last. The very recent history (as discussed in "possibility one" above) is always reachable, fortunately. Once, this synchronization succeeds, all part of the beaconchain is queriable with good latency.

For any kind of remote validator monitoring (typical questions like "how much did a validator lose in the last 3 days due to spurious internet outages"), the only setting that is usable currently is the full archive, but in practice, a partly full archive, which e.g. removes blocks after 2 months (or configurable), would be very useful. This feature would even be useful, if no new synchronization logic were to be implemented, simply the archival would "start" when you set up the non-archive caplin, and then slowly fill up your 2 months (in realtime), and from then on the 2 months are always available.

Alternatively, if some kind of torrent technology is implemented, instead of the very slow "State processing progress", then maybe this whole discussion is moot, as the datasize itself is <200G for the full archive, so the only problem is the slow generation of it. Compared to the size of the execution chain, one can definitely accept if the only feature for archival access is a 200G download and storage, if the initialization of this fast (similarly to the execution chain in 3.0.0-alpha3).

Implementation

I unfortunately don't have enough knowledge of the codebase to know if this is a quick change, but I'm willing to poke around if somebody has ideas, I can definitely do small modifications if directed and I can test+report back.

Also, maybe there is some fundamental reason why what I'm asking is not possible, I just very naively figured that if a data can be securely accessed now (with confidence that it has been verified and it's not a lie), then it's possible to archive it for 2 months. In the complicated world of PoS and crypto in general, there might be some issue with my oversimplified reasoning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants