Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High memory usage with paritydb #26

Closed
liuchengxu opened this issue Jul 21, 2024 · 1 comment
Closed

High memory usage with paritydb #26

liuchengxu opened this issue Jul 21, 2024 · 1 comment

Comments

@liuchengxu
Copy link
Member

liuchengxu commented Jul 21, 2024

It's observed that simply running subcoin with paritydb for the initial Bitcoin full sync could consume up to 50+GB memory when the chain syncs ~400000 blocks on my machine (128 GB in total, Ubuntu 22.04), which is ridiculous. Rocksdb does not have this issue, the peek memory usage of rocksdb is less than 2GB when the chain syncs to the same height. It's a known issue of paritydb (paritytech/parity-db#168) We should find a workaround or disable paritydb and stick with rocksdb before the memory issue is resolved upstream.

@liuchengxu liuchengxu added the bug Something isn't working label Jul 21, 2024
@liuchengxu
Copy link
Member Author

Based on a recent test conducted on a cloud machine with 8 GiB of memory, there doesn't appear to be any issue. The memory usage for Subcoin remained stable at around 35%, and there were no fatal errors such as OOM. It seems that ParityDB only uses a proportion of the available memory, and the usage does not increase indefinitely. Therefore, running with ParityDB should not pose a problem.

@liuchengxu liuchengxu removed the bug Something isn't working label Aug 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant