feat(sequencer): integrate slinky oracle and vote extension logic #1236
Conversation
github-actions
bot
added
proto
| market_map::v1::{ | ||
| GenesisState as MarketMapGenesisState, | ||
| GenesisStateError as MarketMapGenesisStateError, | ||
| }, | ||
| oracle::v1::{ | ||
| GenesisState as OracleGenesisState, | ||
| GenesisStateError as OracleGenesisStateError, | ||
| }, |
There was a problem hiding this comment.
Instead of aliasing the types, can you alias the module instead? Like so:
| market_map::v1::{ | |
| GenesisState as MarketMapGenesisState, | |
| GenesisStateError as MarketMapGenesisStateError, | |
| }, | |
| oracle::v1::{ | |
| GenesisState as OracleGenesisState, | |
| GenesisStateError as OracleGenesisStateError, | |
| }, | |
| market_map::v1 as market_map, | |
| oracle::v1 as oracle, |
Then you can refer to the types as market_map::GenesisState and oracle::GenesisState. I prefer to restrict type aliases to cases like hiding type parameters that would be repeated everywhere (as in the case of anyhow::Result<T> = Result<T, anyhow::Error>.
| @@ -712,6 +910,25 @@ mod tests { | |||
| bridge_sudo_change_fee: Some(24.into()), | |||
| ics20_withdrawal_base_fee: Some(24.into()), | |||
| }), | |||
| slinky: Some( | |||
There was a problem hiding this comment.
Can you fill out the genesis data here? This is used in the snapshot test below and it would be great to catch if this changed.
There is a unit test further down (can't comment on it because it's outside the diff range), mismatched_addresses_are_caught, which could also benefit from a check on the market maker address.
| SlinkyGenesis { | ||
| market_map: MarketMapGenesisState { | ||
| market_map: MarketMap { | ||
| markets: IndexMap::new(), |
There was a problem hiding this comment.
For filling in some data here, there is a nice utility crate that provides hashmap! and btreemap! similar to vec! in the standard library: https://docs.rs/maplit/latest/maplit/
And indexmap itself already provides this itself: https://docs.rs/indexmap/latest/indexmap/macro.indexmap.html
| "admin": "astria1rsxyjrcm255ds9euthjx6yc3vrjt9sxrm9cfgm" | ||
| } | ||
| }, | ||
| "oracle": {} |
There was a problem hiding this comment.
Commented on the file where the test that led to this snap was defined: would be great to populate this with some extra data.
| type Err = CurrencyPairParseError; | ||
|
|
||
| fn from_str(s: &str) -> Result<Self, Self::Err> { | ||
| let re = regex::Regex::new(r"^([a-zA-Z]+)/([a-zA-Z]+)$").expect("valid regex"); |
There was a problem hiding this comment.
Right now this regex would be recompiled on every parse, which is not very efficient!
Have a look at what we are doing inside composer to avoid that at the bottom of this comment. However, don't use once_cell's Lazy or the new std LazyLock. Both of them block their respective thread which would be bad in an async context.
Instead, use https://doc.rust-lang.org/stable/std/sync/struct.OnceLock.html, which does not:
static CURRENCY_PAIR_REGEX: OnceLock<Regex> = OnceLock::new();
fn currency_pair_regex() -> &Regex {
CURRENCT_PAIR_REGEX.get_or_init(|| regex::Regex::new(r"^([a-zA-Z]+)/([a-zA-Z]+)$").expect("valid regex"))
}Also take a look at how we defined the regex in composer, which gives names to the capture groups so its self-documenting. :)
astria/crates/astria-composer/src/rollup.rs
Lines 36 to 60 in 0ed3c19
| } | ||
|
|
||
| #[instrument(name = "MarketMapComponent::end_block", skip(_state))] | ||
| async fn end_block<S: StateWriteExt + 'static>( |
| } | ||
|
|
||
| #[instrument(skip_all)] | ||
| async fn get_currency_pair_mapping(&self) -> Result<HashMap<u64, CurrencyPair>> { |
| let mut currency_pairs = HashMap::new(); | ||
|
|
||
| let mut stream = std::pin::pin!(self.prefix_keys(&prefix)); | ||
| while let Some(Ok(key)) = stream.next().await { |
There was a problem hiding this comment.
This is potentially a source of latency: use futures::TryStreamExt::try_collect on a futures::StreamExt::buffered adaptor to do this concurrently.
| } | ||
|
|
||
| #[instrument(skip_all)] | ||
| async fn get_all_currency_pairs(&self) -> Result<Vec<CurrencyPair>> { |
There was a problem hiding this comment.
As above, implement this as a stream and let the call site decide what to do with this. See #1453 for how to do that.
| @@ -15,6 +17,7 @@ message GenesisAppState { | |||
| IbcParameters ibc_parameters = 8; | |||
| repeated string allowed_fee_assets = 9; | |||
| Fees fees = 10; | |||
| SlinkyGenesis slinky = 11; | |||
There was a problem hiding this comment.
Yeah, I have changed my mind. We should probably call this slinky because there is nothing generic about this. (Or should we use their new *-connect name already?)
|
|
||
| pub(crate) async fn get_max_num_currency_pairs<S: StateReadExt>( | ||
| state: &S, | ||
| is_proposal_phase: bool, |
There was a problem hiding this comment.
This low level function should not be aware of what happens at the higher level, I think. I looked at the is_proposal_phase function and had to follow it all the way here. IMO move this logic to the call-site.
| }) | ||
| } | ||
|
|
||
| pub(crate) async fn verify_vote_extension<S: StateReadExt>( |
There was a problem hiding this comment.
Is it intended that if the oracle is disabled, that this function is still called? If the oracle is disabled, NO_ORACLE=true, I would expect that a sequencer node completely abstains from participating in vote extensions.
I think it might be clearer if we separated the config in
- participates in vote extensions,
- only verifies vote extensions,
- ignores vote extensions?
…1470) Extensions on `cnidarium::StateRead` should implement a stream rather than allocate and collect into datastructures.
This patch enforces more type strictes on slinky oracle domain types: 1. Nonces, Ids, and Prices are new-type wrappers around their respective primtives; This is to avoid using `u128` in place of a `Price`, for example. 2. All oracle types written to state get their own types that themselves derive the borsh serialization traits. 3. All json is removed from the oracle state read and write extension trait. 4. A lot of validation code is moved from sequencer to core so that validation happens closer to the boundary. 5. Functions in the public interface for constructing various validation errors were removed. 6. Constructors that would allow violating invariants (specifically invariants around the contents of currency pairs) were removed. A bug was fixed in the `oracle::StateReadExt::put_price_for_currency_pair` trait method, which was not updating the nonce.
Summary
integrate skip's slinky oracle service into astria.
at a high level, slinky consists of an oracle sidecar program, which interacts with a validator node to provide price data, and various cosmos-sdk modules.
since astria isn't cosmos, the relevant cosmos modules (x/marketmap and x/oracle) were essentially ported into the
slinkymodule of the sequencer app, which consists of two components,market_mapandoracle.the sequencer app was updated to talk to the sidecar during the
extend_votephase of consensus to gather prices to put into a vote extension.the vote extension validation logic, proposal logic, and finalization logic were also ported from slinky.
Background
we want oracle data to be available to rollups (and maybe on the sequencer itself too?)
Changes
market_mapandoraclevaluesmarket_mapcomponent for the sequenceroraclecomponent for the sequencer and the query service for this componentextend_votelogic which gets the prices from the sidecar and turns them into a vote extensionverify_vote_extensionlogic which performs basic validation on a vote extension during the consensus phaseprepare_proposallogic which gathers the vote extensions from the previous block, prunes any invalid votes, and performs additional validation to create a valid set of VEsprocess_proposallogic which validates the set of VEs proposed, checking signatures and that the voting power is >2/3 amongst other thingsfinalize_blocklogic which writes the updated prices to state based on the commited vote extensions. skip uses stake-weighted median to calculate the final price, but we don't have stake-weighting yet, so i just took the median.Msgtypes as sequencer actions (follow-up)SequencerBlockHeaderto contain the extended commit info + a proof for it (also follow-up)DeltaCurrencyPairStrategy- right now only theDefaultCurrencyPairStrategyis implemented. can also do in follow-upTesting
TODO: run this on a multi-validator network also
clone slinky: https://github.com/skip-mev/slinky/tree/main
install go 1.22
build and run slinky:
checkout
noot/slinkybranch of astriarun sequencer app w/ grpc port 9090 and
ASTRIA_SEQUENCER_ORACLE_ENABLED=truerm -rf /tmp/astria_db rm -rf ~/.cometbft rm app-genesis-state.json just run just run-cometbftshould see a sequencer log like:
should see a slinky log like:
{"level":"info","ts":"2024-07-02T14:33:46.318-0400","caller":"marketmap/fetcher.go:147","msg":"successfully fetched market map data from module; checking if market map has changed","pid":727051,"process":"oracle","fetcher":"marketmap_api"}then, when blocks are made, should see logs like the following for each block:
Breaking Changelist