Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incrementalFetch(): fix race condition which would miss rows based on changed_at #288

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Al2Klimov
Copy link
Member

incrementalFetch() relies on every newly inserted row having a higher changed_at than every already committed one. At the moment, web doesn't guarantee this. It even writes changed_at rounded to whole seconds. Hence, we could skip one of two rows which appeared during the same second.

This change simply ignores the most recent 3s of config updates (for 3s), so that two or more new rows which appear during the same second aren't a problem. They're processed together 3s after they get committed.

… changed_at

incrementalFetch() relies on every newly inserted row having a higher changed_at than every already committed one. At the moment, web doesn't guarantee this. It even writes changed_at rounded to whole seconds. Hence, we could skip one of two rows which appeared during the same second.

This change simply ignores the most recent 3s of config updates (for 3s), so that two or more new rows which appear during the same second aren't a problem. They're processed together 3s after they get committed.
@Al2Klimov Al2Klimov requested a review from oxzi March 11, 2025 11:25
@cla-bot cla-bot bot added the cla/signed CLA is signed by all contributors of a PR label Mar 11, 2025
@oxzi
Copy link
Member

oxzi commented Mar 11, 2025

Before you spend any more time looking into this, please take a look at #237.

@Al2Klimov
Copy link
Member Author

Before you spend any more time looking into this,

Don't worry. I was already done. (#288 (comment), #289)

please take a look at #237.

Seems more complicated to me. 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla/signed CLA is signed by all contributors of a PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants