-
Notifications
You must be signed in to change notification settings - Fork 296
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
breakage against Noble testnet v8.0.0-rc.2 #4899
Comments
Based on a research spike by @avahowell in collaboration with Astria, we tried setting
We're surprised because this setting did resolve testnet relaying for Astria, but it hasn't for us. Another possible resolution is bumping the version of
Unclear whether upgrading the tendermint-rs version would constitute a consensus-breaking change. At the very least, we should understand whether bumping the dep resolves the issue we're seeing. |
Paired with @avahowell to investigate the hermes setup. Turns out that despite the logged error messages, hermes does still properly relay packets. The current penumbra testnet has a short unbonding period, which results in short-lived ibc clients (on the order of 20m or so currently). We confirmed that:
The error messages are unfortunate, but also present on the penumbra/osmosis testnet service, which also uses cometbft v0.38.x on the counterparty side. We should rebase Hermes on latest upstream main, but that work should be tracked separately. We're also investigating a plan to publish the Penumbra workspace crates to crate.io, to support upstreaming the Penumbra config into hermes. Still unresolved is the grpc problem that originally motivated this ticket. But as for the potential of breakage when Noble v8 is released, it appears that hermes operators should at |
Describe the bug
An upcoming Noble chain upgrade to v8 is being prepared on the Noble testnet. For the Penumbra Labs testnet (https://testnet.plinfra.net), we've been running a version of Hermes that relays between
penumbra-testnet-phobos-2
andgrand-1
. On or around 2024-10-17, we started observing breakage when communicating with the Noble testnet node endpoint run by Polkachu:noble-testnet-grpc.polkachu.com:21590
I confirmed out of band with Polkachu that this breakage corresponded to deployment of the https://github.com/noble-assets/noble/releases/tag/v8.0.0-rc.2 tag to the testnet endpoint.
We first discovered this breakage when testing the behavior of the diff in #4878. Similar breakage is also evident in the hermes relayer that PL is running.
Example error messages
When running on the feature branch for #4878:
When viewing the logs for the hermes relayer instance between testnets:
Additional context
The gRPC endpoint is at least functional enough to return service descriptors:
We also know that the cometbft rpc is returning structured data:
Although we should be careful to determine how the structure violates assumptions in the code, given the parse error.
The text was updated successfully, but these errors were encountered: