Add main slippage query

[harisang]

This PR splits the main slippage accounting query into two queries:

1. The Processed Token Imbalances query (4057345), which computes the token imbalances per tx. Here, we use the term "processed" since we subtract network and protocol fees from the imbalances (i.e., it is not just the raw token imbalances). On the way, we remove some code that was not used anymore, mainly around internalized interactions.
2. The main query (427730) now uses 4057345 as black box to fetch imbalances and then creates a Dune price feed to convert these imbalances to ETH.

The excluded batches query (3490353) was already a separate query, which we now put here under version control

[harisang]

I am not sure how to fix the sqlfluff errors around the use of the dictionary from the cowswap.raw_order_rewards table, so any help would be appreciated.

[fhenneke]

Is this an execution speed optimization or an artifact from removing parts of the query?

[harisang]

Artifact of removing part of the query. The union above was written as

select * from (select * from .... union select * from ....) as _ order by tx_hash

and sqlfluff was complaining about the underscore

[fhenneke]

Then it can be removed and the table all_transfers_temp can be used directly below.

[fhenneke]

This looks like it (and lots of the code above) could be replaced by

select
    tx_hash,
    sender,
    receiver,
    token_address as token
from "query_4021257(blockchain='ethereum',start_time='{{start_time}}',end_time='{{end_time}}')"

[harisang]

i was not at all aware of this query. I will need to look at it and see what i can do

[fhenneke]

To me this table should be the result of an independent query. It looks super close to what query 4021257 does, with a sprinkle of joining on batch and erc20 data and adding a sign.

What is the use of joining with additional data here?

[fhenneke]

This table could also just have columns block_time, tx_hash, token, amount(, transfer_type for debugging). It could be constructed from the result of two queries:

• token imbalances; where transfer_type in ('user_in', 'user_out', 'amm_in', 'amm_out')
• fees; where transfer_type in ('protocol_fee', 'network_fee'(, 'partner_fee'))

[fhenneke]

Truncating to hour here seems strange. The final token balance sheet might as well only have columns block_time, tx_hash, token, amount

[harisang]

yeah definitely a good point

[harisang]

Removed the truncation for now

[fhenneke]

I do not thing this table has an index (anymore?).

[fhenneke]

this parameter seems unnecessary

[harisang]

this is so that we can also see the results_per_tx table, if needed.

[fhenneke]

the results per transaction table should probably be its own query. or rather there should be a query with all balances changes (also those which cancel) and prices.

the aggregation and printing of links seems conceptually different. the filtering on excluded batches should also probably happen between results_per_tx and results as it is mostly related to prices and not to missing information on imbalances.

if those queries are split, the first query could also be used to compute unusual slippage settlements

[harisang]

Did some changes to address some of these.

[fhenneke]

there cannot be a new line after with (unless it is a comment)

[fhenneke]

One alternative approach would be to have a query like this with a balance sheet per transaction per token with prices. This is typically the level I am looking at when debugging slippage. (Sometimes I go down one level deeper and want to see classified balance changes with prices.)

That can then be aggregated into values per batch and values per solver.

[harisang]

The latest version was tested and is able to give identical results to the current query for the accounting week of Sep3-10, 2024.

I am taking the libery to merge this as we need to have a working version for tomorrow's payouts. And i would suggest a new PR that properly restructures slippage accounting