Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: go 1.22.x and pebble v1.1.0 #3203

Closed
wants to merge 3 commits into from

Conversation

faddat
Copy link

@faddat faddat commented Feb 22, 2024

  • bump deps
  • move to local machine: updating pyroscope
  • Add Grafana profiling libraries

Summary

This PR allows Celestia to use pebble, a kv store from cockroachdb that is like Facebook's RocksDB except written in native go. It makes celestia-appd much more performant, and might make celestia-node faster too, though that is still yet to be seen and understood.

It's supported by or related to the following pull requests:

In total, the goal of this pull request is to ensure that Celestia doesn't "run out" of keys in the kv store -- goleveldb does this, and so does pebble, but it happens much, much later with pebble. Operators of celestia-appd who want archival state will find this set of PR's particularly appealing.

celestia-core

I think that these changes are especially important to Celestia core. The reason for that is that in fact, the changes to the database library, allow for a much much simpler build process for celestial core, one that does not even need darker, and one that is many times faster. I've been trying to contribute it to tendermint or comet for over two years, but it hasn't worked out for whatever reason.

I'm going to migrate these changes over to celestia core, as the complex docker based build process seen in cometbft is simply unnecessary, and we can make CI vastly faster while speeding up developer onboarding.

In a very real way, getting this series of PRs landed is a first step to that, as the version of cometbft-db seen here doesn't even have a Dockerfile in the repository.

@github-actions github-actions bot added the external Issues created by non node team members label Feb 22, 2024
@Wondertan
Copy link
Member

Node doesn't use the cometbft-db, so bumping core with pebble won't give any meaningful benefits, and this PR is technically just a regular dependency bump.

As it's a bump to a dangling commit, I am closing this PR and we can return to bumping this through our release process.

@Wondertan Wondertan closed this Feb 22, 2024
@Wondertan
Copy link
Member

Btw, node uses badgerdb for the KVBackend, and if configured well, it can yield faster write performance than pebble. Badger's defaults are horrible though

@faddat
Copy link
Author

faddat commented Feb 22, 2024

@Wondertan I haven't seen badger beat pebble (ever)

About the rest of the PR, your assessment makes sense.

Another thing this does, however, is unblock go 1.22

I'm not totally sure how badger does when faced with more keys than the iterator can seek through efficiently?

that's kinda the problem I'm aiming at here.

For celestia-core, the goal is to get rid of the needless cgo-oriented docker build process.

@Wondertan
Copy link
Member

I've seen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
external Issues created by non node team members
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants