Skip to content
This repository has been archived by the owner on Dec 20, 2024. It is now read-only.

Delays when syncing data #72

Closed
harisang opened this issue Jan 27, 2024 · 1 comment
Closed

Delays when syncing data #72

harisang opened this issue Jan 27, 2024 · 1 comment

Comments

@harisang
Copy link
Contributor

After PR #71, which fixed the issue with the backend database change, it seems to me that we might sync data with signficant delay, as the syncing, I believe, now depends on both prod and staging having at least one settlement every few hours. The reason for that, I think, is the min operator in this line:

return min(int(barn["latest"][0]), int(prod["latest"][0])) - REORG_THRESHOLD

I would propose that we actually change it to max now. Let me know what you think @fhenneke

@fhenneke
Copy link
Contributor

fhenneke commented Jan 27, 2024

I would change it to max and remove the - REORG_THRESHOLD.

With the new query we look at blocks which have already successfully been processed (since these is an entry in settlement_observations). What can then go wrong is that maybe there are staging and production settlements on adjacent blocks and the later settlement is already processed but not the earlier one. Then the earlier one will not end up on dune. Not sure if that is actually happening sometimes. For that purpose we could have some delay, so something like - MAX_PROCESSING_DELAY with MAX_PROCESSING_DELAY = 1 or 2 or whatever seems fit. The old query had a delay of 65 which might included implicitly a MAX_PROCESSING_DELAY = 1 since the reorg threshold should be 64.

fhenneke added a commit that referenced this issue Feb 5, 2024
This PR addresses issue #72. At the moment the block range is computed
using the minimum of the latest finalized block from production and
staging and subtracting `REORG_THRESHOLD = 65`. This was done as a hot fix in #71.
One problem with this approach is that having few settlements in staging
blocks upload of production data to dune.

This PR changes the behavior to look at the maximum block of the latest
finalized blocks from production and staging subtracting
`MAX_PROCESSING_DELAY = 1`. This should leave enough time for the autopilot
to process settlements after finalization in staging and production.
@fhenneke fhenneke closed this as completed Feb 9, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants