[Epic] Solver Slippage V2

[bh2smith]

Based on Specification Documentation

We layout a top-down approach plan for this repo. Development will take place in a "Feature Branch" and this is an open PR containing an aggregation of all changes.

Phase 1: Acquiring Internalized Transfers

Note that a Proof of Concept has already been implemented in python here and can be used for reference when reviewing (cc @gentrexha)

• Setup Initial DB Migration (Add initial DB migration scripting #178)
• Basic Tenderly Web3 Action For Settlement Events ([Easy] Basic Web3 Action for Settlements #181)
• Add Models & Parsing Logic to WebHook (Action Models #182)
• Parse Trades and Transfers from Settlement Transaction Receipts. Parse event logs with ethers.js #193
• Use Action to Insert SettlementEvents to DB (Write Settlements to DB from Action #185)
• Fetch winning settlement from orderbook API by txHash (Fetch: Winning Settlement from Orderbook API #220)
• Simulate Full Call Data from Winning Settlement (Simulate: Transaction via Tenderly #227)
• Parse Simulation Results (Parse: Tenderly Simulation Response #228)
• Parse transfer events from simulated transaction (Parse: Partition Settlment Events from Sim Logs #229)
• Aggregate transfers with accumulator as "imbalance" ([Easy] Aggregate Transfers #233)
• Token Imbalance Arithmetic ([Easy] Token Imbalance Arithmetic #234)
• Project Milestone Compute Internalized Imbalance as Full - Actual (🔑 - Get Internalized Imbalance #231)
• Insert InternalizedImbalances into DB (Create Table internalized_imbalances #242)
• Tie together all functionality as a single workflow to call from within the web-hook (Full Pipeline #261)
• Introduce Data dumping ground for unconfirmed transactions ([Database] Tx Receipts #271)
• Adjust pipeline to account for finalized and unconfirmed transactions (Preliminary & Primary Pipeline Functionality #272)
• Extend Tx Receipt so that Settlement Log index is block-wise not transaction-wise. ([Bug] Log Index Parsing #275)
• Mark Processed (Insert Settlement & Mark Processed #276)
• Deploy to Staging DB (announcement)
• Bug - [Bug] Catch Missing Competition #281
• Bug - Try More Simulation Blocks and Write Default on Failure #279
• MAIN BRANCH CONCLUDING PHASE 1 ([Epic] Slippage Accounting - Phase I #184)

Phase 2: Dune Sync

• Obtain a Production DB and Deploy Phase 1 to production (slack)
• Configure auto deployments.
• Inject data into dune community sources. This is part of a different project -- namely dune-sync ([Slippage V2] Sync Internal Transfers dune-sync#37)
• This project will need to be able to to determine if there is any missing data before sync.
• If something is missing, it should be able to trigger the pipeline "en masse" (for a collection of txHashes).

Phase 3: Refactor Slippage Query

• Replace this portion of the slippage query with internalized transfer data. ([Slippage] Account for Phantom Transfers #259)

Additional TODOs & Cleanup

• Ensure Correct Solver Address for Winning Settlement #226
• Remove txHash from SimulationData #230
• Be cautious of Solver Address from Competition Table #232
• Handle Null Logs  #235
• Use Fixed Type for EVM Addresses #236
• Recover Native Asset (ETH) Transfers #237
• Account for ETH (un)wraps #246
• Alternative Simulation Implementation #245
• Manual mass application of Phase 1 (i.e. without web-hook trigger) and ensure idempotency. ([Trivial] A few cosmetic changes #244)
• Complete E2E Test of Pipeline #274

Follow Up Tasks

• Migrate primary functionality of Phase 1 into an AWS lambda and deploy. This will reduce attack surface by not exposing DB credentials to a third party service. Will also give us a gateway to more involved pipeline projects. Prototype exists here
• ... probably more stuff.

[bh2smith]

Some very helpful Proof of Concept Code can be found

• A Typed Python Implementation of the full pipeline excluding DB writes.
• Components of the Grand Scheme including Tenderly Action that writes to DB & invokes AWS lambda.

[fleupold]

I think one open design decision is what data we should store in our database (future data warehouse). I believe @olgafetisova was in favor of ingesting the raw transaction receipts/simulation results and then run the slippage computation based on another script on our database:

• Tenderly Simulation Result --> DB --> Script to compute slippage --> DB (or unmaterialized view)
• Tenderly Simulation Result --> Script to compute slippage --> DB

The latter has the advantage IMO that it would make the script easily reproducible by third parties (all they need is tenderly access, no need to access our DB). On the other hand, having one script ingesting the simulation result and another script computing the slippage based off that splits responsibilities and may allow us more easily replace the tenderly dependency (e.g by our own node simulation) later on. Also, if there are other use cases that would use those simulation results, we wouldn't have to run the simulation multiple times.

[bh2smith]

Thanks Felix, these are all very good points to consider and the avenue I have been going is definitely the second. I want to point out that you keep mentioning "Script to compute slippage" as if that is what we are doing here... however this project is aimed at computing "internalized token imbalance". If that is not your understanding, then this should be aligned (maybe you want to read the introduction section of the specification). We can easily change gears if necessary, but the goal here currently is to replace the approximation being made in the existing slippage query. Long term, we would transition this into a fully internal slippage computation.

raw transaction receipts/simulation results

I am very much in favour of saving simulations and storing their IDs on our side, but each simulation is a VERY LARGE JSON blob that I don't think we should be keeping on our side (at least not without reducing it by pruning unnecessary parts).

replace tenderly dependency

The project has been written in a way (with an interface TxSimulator -- cf #227) that should already make it easy enough to swap out our TenderlySimulator

[fleupold]

you keep mentioning "Script to compute slippage" as if that is what we are doing here...

I was judging from the title of this EPIC ("Solver Slippage v2"), that the final deliverable of this EPIC was a script to compute more accurate solver slippage. On the way there the goal is to pioneer and build the foundation for our data warehouse infra.

[bh2smith]

Foreseeable Issues & Complications

1. This "uninternalizedCallData" which this project requires may not be accessible upon solver-driver colocation. This would essentially kill this project (in ~2-3 months)
2. Going back in time to recover historical data will likely result in rate limiting.

Workflow Failure Points

• Each settlement requires a passing simulation to yield "correct" results. When simulation fails? Try simBlock + 1? Then what? We could treat it as if there were no internalized transfers and "let it slide"
• What happens when the array of solutions has length zero?
• what to do when the actual settlement solver does not agree with the winning solver from the competition?

[bh2smith]

final deliverable of this EPIC was a script to compute more accurate solver slippage

Yes this is correct, but does not mention computing it from our own database. Rather an adaptation of our existing query.

On the way there the goal is to pioneer and build the foundation for our data warehouse infra.

In fact this is already happening. Its just that we are not just gonna start pumping a bunch of bloat into a database right from the start. It will be very small and compact representation of what we need to "compute accurate slippage"

[olgafetisova]

As agreed earlier with Ben, he is going to give us a presentation with the choices that have been already made and the actual state of the project, additionally, he is going to host a discussion session on the open questions so that all the reviewers of the project are aligned on the next steps.