sync2: implement database-backed sync based on FPTree #6406
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ivan4th
requested review from
dshulyak,
fasmat,
poszu,
acud and
jellonek
as code owners
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## sync2/sqlstore #6406 +/- ##
================================================
Coverage 79.8% 79.9%
================================================
Files 342 350 +8
Lines 44310 45818 +1508
================================================
+ Hits 35391 36613 +1222
- Misses 6920 7128 +208
- Partials 1999 2077 +78 ☔ View full report in Codecov by Sentry. |
ivan4th
force-pushed
the
sync2/sqlstore
branch
from
f886141
to
bb43161
Compare
`fptree.FPTree` provides a sufficiently efficient data structure for performing range fingerprinting on data residing in database tables, speeding up the queries at expense of some memory use. `dbset.DBSet` builds on `fptree.FPTree` and provides a database-backed implementation of the `multipeer.OrderedSet` interface.
## Motivation Ensure an active statement is reset even if `decoder` in `db.Exec` panics
…6403) ## Motivation To detect errors that would break compatibility with previous versions of the node I extended `TestSmeshing` to deploy 25% of the nodes in the test using an older version of go-spacemesh. So if any change changes protocol rules that would break consensus like the error fixed in #6398 or the error fixed in #5839.
## Motivation In rare cases where building the initial ATX times out the initial PoST stored in the local DB is deleted. This leads to an endless loop of the node not being able to publish its first ATX because it expects an initial PoST but never generates one. This PR fixes this issue.
## Motivation All tests have been updated to ensure that DB connections are closed at the end of the test. This ensures that any open connections that weren't returned to the pool at the end of the test cause it to fail and would otherwise go unnoticed.
## Motivation Given that after recent item sync is done (if it's needed at all), the range set reconciliation algorithm no longer depends on newly received item being added to the set, we can save memory by not adding the received items during reconciliation, except for the recent items
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The new set reconciliation mechanism requires an efficient data structure for handling range fingerprint queries.
Description
fptree.FPTreeprovides a sufficiently efficient data structure forperforming range fingerprinting on data residing in database tables,
speeding up the queries at expense of some memory use.
dbset.DBSetbuilds onfptree.FPTreeand provides a database-backedimplementation of the
multipeer.OrderedSetinterface.#6405 needs to be merged before this one.