-
Notifications
You must be signed in to change notification settings - Fork 49
Incremental Watched Queries #614
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
🦋 Changeset detectedLatest commit: c55dc01 The changes in this PR will be included in the next version bump. This PR includes changesets to release 9 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces an incremental watched query system across the PowerSync platform, replacing the legacy table-change–based watches with comparison- and differential-based modes, and refactors the web, Vue, and React packages (including their tests and demos) to adopt the new API.
- Adds
incrementalWatch
API toAbstractPowerSyncDatabase
withCOMPARISON
andDIFFERENTIAL
modes - Implements
WatchedQuery
, processors, and builders in@powersync/common
- Updates Vue and React composables/hooks/tests to consume the new incremental watch API
Reviewed Changes
Copilot reviewed 86 out of 88 changed files in this pull request and generated 5 comments.
Show a summary per file
File | Description |
---|---|
packages/web/tests/multiple_instances.test.ts | DRYed stream instantiation; increased retry delay |
packages/web/src/worker/sync/SharedSyncImplementation.ts | Hardened _testUpdateAllStatuses , removed no-op code |
packages/web/src/db/adapters/LockedAsyncDatabaseAdapter.ts | Added abort‐on‐close logic and lock timeout support |
packages/vue/src/composables/useWatchedQuery.ts | New hook using incrementalWatch |
packages/vue/src/composables/useSingleQuery.ts, useQuery.ts | Refactored to share logic between single/watch hooks |
packages/vue/tests/*.test.ts | Switched to real PowerSyncDatabase and plugin setup |
packages/react/src/hooks/watched/watch-utils.ts | New query-compatibility helpers |
packages/react/src/hooks/watched/* | New useWatchedQuery , useSingleQuery , subscription |
packages/react/src/index.ts | Exports updated hooks |
packages/common/src/client/watched/*.ts | WatchedQuery , processors, comparators, builders |
packages/common/src/utils/BaseObserver.ts, DataStream.ts | Observer enhancements, error event support |
AbstractPowerSyncDatabase incrementalWatch and onChange updates | New entry point for incremental queries |
packages/common/src/client/watched/processors/ComparisonWatchedQueryBuilder.ts
Outdated
Show resolved
Hide resolved
demos/react-supabase-todolist/src/components/providers/SystemProvider.tsx
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Happy with the changeset, while massive. Well documented and structured.
const watchedQuery = this.incrementalWatch({ mode: IncrementalWatchMode.COMPARISON }).build<QueryResult | null>({ | ||
comparator, | ||
watch: { | ||
query: { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This formal query definition works, but a shorthand alternative makes for the query option that takes in the SQL/Params and always does an executeRead/getAll by default/implicitly would be nice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do understand the sentiment behind this concern. I also thought about this and introduced the GetAllQuery
helper for this reason.
It is currently possible to supply a GetAllQuery
as the query
option e.g.
const watch = powersync
.incrementalWatch({
mode: IncrementalWatchMode.COMPARISON
})
.build({
watch: {
query: new GetAllQuery({
sql: 'SELECT * FROM assets',
parameters: []
}),
placeholderData: []
}
});
Which is slightly more typing that just supplying an object of the form {sql: string, parameters?: any[]}
, but IMO provides a better experience than the alternatives like:
Having query
be an union type of {sql: string, parameters?: any[]} | WatchCompatibleQuery
.
This gives bad autocomplete (IMO) if a dev was trying to create this option (it gives all possible options as a first suggestion).
Alternatively, a Discriminated Union type would also not be directly compatible with our current CompatibleQuery
interface.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a big PR, so just giving some high-level comments to start with:
Internals & Hooks
I really like the refactoring to move more of the logic into the internals - it's nice that the hooks are now fairly thin wrappers around the internal methods.
incrementalWatch
The db.incrementalWatch
API feels a little difficult to follow with the current structure. It feels like the API changes completely based on which mode you select, and should perhaps rather be two separate APIs?
Another option is to invert the API to start with the query part first, for example something like this:
db.createQuery(sql, params).differentialWatch(...)
This can avoid the explosion of new methods on the database itself, instead having the different methods on a new Query-type class. You could perhaps also have different methods for creating this from different sources, e.g. to create from a Kysely or Drizzle query.
Incremental comparison queries
I see the main benefit of these to avoid re-renders, to improve rendering performance. It would be good to see some benchmarks to validate that this is worth the additional complexity, that we're not just moving overhead from the rendering part to the query part.
I also wonder if we should unify comparitors and differentiators, always using the differentiator API (identify + compareBy instead of just compare). This way, you can do the re-rendering on an even more granular level. So if you have 1 row out of 1000 results that changed, you can keep the other 999 row objects from the previous results, avoiding further re-renders on those rows. Once again, we'd need some form of benchmarks to check that this is actually worth it.
* A hook to access and subscribe to the results of an existing {@link WatchedQuery} instance. | ||
* @example | ||
* export const ContentComponent = () => { | ||
* const { data: lists } = useWatchedQuerySuspenseSubscription(listsQuery); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The name is wrong in this example.
if (signal.aborted) { | ||
return; // Abort if the signal is already aborted | ||
} | ||
console.log('done with query refresh'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should remove these log statements (including the one above)
…ces better in differential queries.
…fferentiator implementation.
Update: I've updated the implementation to guard the new watch methods behind the new Comparison based queries are now accessed via The The An automated React profiling benchmark suite has been added in |
Overview
Our current Watched query implementations emit results whenever a change to a dependant SQLite table occurs. The table changes might not affect the query result set, but we still query and emit a new result set for each table change. The result sets typically contain the same data, but these results are new Array/object references which will cause re-renders in certain frameworks like React.
This PR overhauls, improves and extends upon the existing watched query implementations by introducing incremental watched queries.
Incrementally Watched Queries can be constructed with varying behaviour. This PR introduces the concept of comparison and differential watched queries.
Comparison based queries behave similar to standard watched queries. These queries still query the SQLite DB under the hood on each dependant table change, but they compare the result set and only incrementally yield results if a change has been made. The latest query result is yielded as the result set.
Differential queries watch a SQL query and report detailed information on the changes between result sets. This gives additional information such as the
added
,removed
,updated
rows between result set changes.Implementation
The logic required for incrementally watched queries requires additional computation and introduces additional complexity to the implementation. For these reasons a new concept of a
WatchedQuery
class is introduced, along with a newincrementalWatch
method allows building a instances ofWatchedQuery
s.The
incrementalWatch
method serves as an entry point to building the varying types of incremental watched queries. Theoptions
to construct different incremental queries vary per themode
desired. Splitting the instantiation into two steps makes specifying specific options cleaner and easier to Type in TypeScript.The
listsQuery
is smart, it:updateSchema
WatchedQuery
instances retain the latest state in memory. SharingWatchedQuery
instances can be used to introduce caching and reduce the number of duplicate DB queries between components.The incremental logic is customisable. Diff based queries can specify custom logic for performing diffs on the relevant data set. By default a
JSON.stringify
approach is used. Different data sets might have more optimal implementations.Updates to query parameters can be performed in a single place, affecting all subscribers.
Reactivity
The existing
watch
method and Reactivity packages have been updated to use incremental queries with differentiation defined as an opt-in feature (defaults to no changes).New hooks have also been added to use shared
WatchedQuery
instances.The Vue and React hooks packages have been updated to remove duplicate implementations of common watched query logic. Historically these packages relied on manually implementing watched queries based off the
onChange
API in order to cater for exposing additional state and custom query executors. The newWatchedQuery
APIs now support all the hook packages' requirements - this effectively reduces the heavy lifting in reactivity packages.The
React Supabase Todolist
demo has been updated with some best practices for reducing re-renders using comparison based incrementally watched queries.Differential Queries
These watched queries report the changes between result sets. The
data
member of aWatchedQuery
'sstate
is of the formA common use case for this is processing newly created items as they are added. The
YJS React Supabase Text Collab
demo has been updated to take advantage of this feature. Document updates are watched via a differential incremental query. New updates are passed to YJS for consolidation as they are synced - the application code no longer needs to track which items were already passed to YJS.