feat(sequencer): integrate connect oracle and vote extension logic

[noot]

Summary

integrate skip's connect (formerly named slinky) oracle service into astria.

at a high level, connect consists of an oracle sidecar program, which interacts with a validator node to provide price data, and various cosmos-sdk modules.

since astria isn't cosmos, the relevant cosmos modules (x/marketmap and x/oracle) were essentially ported into the connect module of the sequencer app, which consists of two components, market_map and oracle.

the sequencer app was updated to talk to the sidecar during the extend_vote phase of consensus to gather prices to put into a vote extension.

the vote extension validation logic, proposal logic, and finalization logic were also ported from connect.

Background

we want oracle data to be available to rollups (and maybe on the sequencer itself too?)

Changes

• import relevant protos from connect and create native rust types for them
• update the sequencer genesis state to contain market_map and oracle values
• implement the market_map component for the sequencer
  • update the sequencer grpc service to support the market_map grpc service, which is required by the oracle sidecar to retrieve the market map from the chain
• implement the oracle component for the sequencer and the query service for this component
• implement extend_vote logic which gets the prices from the sidecar and turns them into a vote extension
• implement verify_vote_extension logic which performs basic validation on a vote extension during the consensus phase
• implement prepare_proposal logic which gathers the vote extensions from the previous block, prunes any invalid votes, and performs additional validation to create a valid set of VEs
• implement process_proposal logic which validates the set of VEs proposed, checking signatures and that the voting power is >2/3 amongst other things
• implement finalize_block logic which writes the updated prices to state based on the committed vote extensions. skip uses stake-weighted median to calculate the final price, but we don't have stake-weighting yet, so i just took the median.
• TODO: implement the connect cosmos Msg types as sequencer actions (follow-up)
  • https://github.com/skip-mev/slinky/blob/158cde8a4b774ac4eec5c6d1a2c16de6a8c6abb5/proto/slinky/oracle/v1/tx.proto
  • https://github.com/skip-mev/slinky/blob/158cde8a4b774ac4eec5c6d1a2c16de6a8c6abb5/proto/slinky/marketmap/v1/tx.proto
• TODO: update SequencerBlockHeader to contain the extended commit info + a proof for it (also follow-up)
• TODO: implement the DeltaCurrencyPairStrategy - right now only the DefaultCurrencyPairStrategy is implemented. can also do in follow-up

Testing

TODO: run this on a multi-validator network also

clone connect: https://github.com/skip-mev/connect/tree/main

install go 1.22

build and run connect:

make build
go run scripts/genesis.go --use-coingecko=true --temp-file=markets.json
./build/connect --market-config-path markets.json --port 8081

checkout noot/slinky branch of astria

run sequencer app and ASTRIA_SEQUENCER_NO_ORACLE=false in .env:

rm -rf /tmp/astria_db
rm -rf ~/.cometbft
just run
just run-cometbft

should see a sequencer log like:

astria_sequencer::sequencer: oracle sidecar is reachable

should see a connect log like:

{"level":"info","ts":"2024-07-02T14:33:46.318-0400","caller":"marketmap/fetcher.go:147","msg":"successfully fetched market map data from module; checking if market map has changed","pid":727051,"process":"oracle","fetcher":"marketmap_api"}

then, when blocks are made, should see logs like the following for each block:

2024-07-05T02:49:21.254163Z DEBUG handle_request:handle_process_proposal: astria_sequencer::service::consensus: proposal processed height=28 time=2024-07-05T02:49:19.143352683Z tx_count=3 proposer=7BE21CDEB6FDCC9299A51F44C6B390EA990E88CD hash=7rmdhOsaW2a0NCZUwSE5yqt2AVR3cOPgGGb4Bb0kpRM= next_validators_hash=F6N7YDQZKfXQld95iV0AmQKNa8DiAxrDnTAcn323QSU=
2024-07-05T02:49:21.310218Z DEBUG handle_request:handle_extend_vote:App::extend_vote: astria_sequencer::app::vote_extension: got prices from oracle sidecar; transforming prices prices_count=118
2024-07-05T02:49:21.323262Z DEBUG handle_request:handle_extend_vote:App::extend_vote: astria_sequencer::app::vote_extension: transformed price for inclusion in vote extension currency_pair="BTC/USD" id=0 price=5683583007
2024-07-05T02:49:21.326070Z DEBUG handle_request:handle_extend_vote:App::extend_vote: astria_sequencer::app::vote_extension: transformed price for inclusion in vote extension currency_pair="ETH/USD" id=1 price=3055069469
2024-07-05T02:49:21.384266Z DEBUG handle_request:finalize_block:App::finalize_block: astria_sequencer::app::vote_extension: applied price from vote extension currency_pair="BTC/USD" price=5683583007 hash=EEB99D84EB1A5B66B4342654C12139CAAB7601547770E3E01866F805BD24A513 height=28 time=2024-07-05T02:49:19.143352683Z proposer=7BE21CDEB6FDCC9299A51F44C6B390EA990E88CD
2024-07-05T02:49:21.384553Z DEBUG handle_request:finalize_block:App::finalize_block: astria_sequencer::app::vote_extension: applied price from vote extension currency_pair="ETH/USD" price=3055069469 hash=EEB99D84EB1A5B66B4342654C12139CAAB7601547770E3E01866F805BD24A513 height=28 time=2024-07-05T02:49:19.143352683Z proposer=7BE21CDEB6FDCC9299A51F44C6B390EA990E88CD

Breaking Changelist

• the PR adds a new proposer-added transaction at
  the start of the block only if vote extensions are enabled. then, there will be 3 special "txs" expected when
  before there were only 2. however if vote extensions are disabled, this won't make a difference.
• the genesis was updated with a new optional field connect, however as this field is optional, it is non-breaking with existing networks.
• if the connect genesis field is set, the sequencer state will change, as the genesis state changes and new values are stored in state. however this does not affect syncing existing networks, as the genesis of the existing network can be used as-is.
• additionally, vote extension participation is optional - if <=2/3 validators participate, blocks are still finalized, just no new oracle data will be published.

[quasystaty1]

nit: Should move under else statement marking genesis changes.

[quasystaty1]

nit: should move under else statement, marking breaking changes. You will have to add the closure

[noot]

is this breaking since it's not adding a new value? what's the correct syntax for this? do i move the whole consensus_params block into the else statement?

[quasystaty1]

Should this value be configurable, at least the port?
Also should move new env vars to else statement.

[quasystaty1]

same here for configurable values

[quasystaty1]

Should move values to mark breaking changes and have new values configurable with value file instead of constants in configmap.