Introduce new logic for syncing raw batch data #140
Conversation
|
New dependencies detected. Learn more about Socket for GitHub ↗︎
|
src/fetch/orderbook.py
Outdated
| .replace("{{start_block}}", str(block_range.block_from)) | ||
| .replace("{{end_block}}", str(block_range.block_to)) | ||
| .replace( | ||
| "{{EPSILON_LOWER}}", "10000000000000000" |
There was a problem hiding this comment.
The value of the cap needs to be adapted for Gnosis and Arbitrum One
There was a problem hiding this comment.
Oh very good point!
So currently it's a very simple strategy. Basically if it is the first day of the month, just rerun the last month as well. Here it assumes that the script will always run at least once every 24h. Of course, this ideally should be fixed, and i would like to add a runtime argument to specify the starting month as well, in case we want to run it again for several months |
| + os.environ.get("NETWORK", "mainnet") | ||
| ) | ||
| else: | ||
| db_url = os.environ[f"{db_env}_DB_URL"] |
There was a problem hiding this comment.
This PR changes the meaning of the PROD_DB_URL and BARN_DB_URL env variables, and this else statement just ensures that it defaults to the "old" way, so that the tests won't fail. This should be removed once the secrets in the repo are edited.
This PR introduces an alternative way of syncing raw batch data on Dune, by relying on the dune api for uploading data as csv files.
The main ideas:
batch_datatable!To test it, one can run
python -m src.main --sync-table batch_dataand check the results by running the following query on Dune:
select * from dune.cowprotocol.dataset_test_batch_rewards_ethereum_2024_11 order by block_deadline DESC