Skip to content

Incremental Watched Queries #614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 84 commits into
base: main
Choose a base branch
from
Open

Incremental Watched Queries #614

wants to merge 84 commits into from

Conversation

stevensJourney
Copy link
Collaborator

@stevensJourney stevensJourney commented May 30, 2025

Overview

Our current Watched query implementations emit results whenever a change to a dependant SQLite table occurs. The table changes might not affect the query result set, but we still query and emit a new result set for each table change. The result sets typically contain the same data, but these results are new Array/object references which will cause re-renders in certain frameworks like React.

This PR overhauls, improves and extends upon the existing watched query implementations by introducing incremental watched queries.

Incrementally Watched Queries can be constructed with varying behaviour. This PR introduces the concept of comparison and differential watched queries.

Comparison based queries behave similar to standard watched queries. These queries still query the SQLite DB under the hood on each dependant table change, but they compare the result set and only incrementally yield results if a change has been made. The latest query result is yielded as the result set.

Differential queries watch a SQL query and report detailed information on the changes between result sets. This gives additional information such as the added, removed, updated rows between result set changes.

Implementation

The logic required for incrementally watched queries requires additional computation and introduces additional complexity to the implementation. For these reasons a new concept of a WatchedQuery class is introduced, along with a new incrementalWatch method allows building a instances of WatchedQuerys.

The incrementalWatch method serves as an entry point to building the varying types of incremental watched queries. The options to construct different incremental queries vary per the mode desired. Splitting the instantiation into two steps makes specifying specific options cleaner and easier to Type in TypeScript.

// Create an instance of a WatchedQuery.
const listsQuery = db
  .query<EnhancedListRecord>({
    sql: /* sql */ `
      SELECT
        ${LISTS_TABLE}.*,
        COUNT(${TODOS_TABLE}.id) AS total_tasks,
        SUM(
          CASE
            WHEN ${TODOS_TABLE}.completed = true THEN 1
            ELSE 0
          END
        ) as completed_tasks
      FROM
        ${LISTS_TABLE}
        LEFT JOIN ${TODOS_TABLE} ON ${LISTS_TABLE}.id = ${TODOS_TABLE}.list_id
      GROUP BY
        ${LISTS_TABLE}.id;
    `
  })
  .differentialWatch();

The listsQuery is smart, it:

  • Automatically reprocesses itself if the PowerSync schema has been updated with updateSchema
  • Automatically closes itself when the PowerSync client has been closed.
  • Allows for the query parameters to be updated after instantiation.
  • Allows shared listening to state changes.
// The registerListener method can be used multiple times to listen for updates.
// The returned dispose function can be used to unsubscribe from the updates.
const disposeSubscriber = listsQuery.registerListener({
  onData: (data) => {
    // This callback will be called whenever the data changes.
    // The data is the result of the executor.
    console.log('Data updated:', data);
  },
  onStateChange: (state) => {
    // This callback will be called whenever the state changes.
    // The state contains metadata about the query, such as isFetching, isLoading, etc.
    console.log(
      'State changed:', 
      state.error, 
      state.isFetching, 
      state.isLoading, 
      state.data
    );
  },
  onError: (error) => {
    // This callback will be called if the query fails.
    console.error('Query error:', error);
  }
});

WatchedQuery instances retain the latest state in memory. Sharing WatchedQuery instances can be used to introduce caching and reduce the number of duplicate DB queries between components.

The incremental logic is customisable. Diff based queries can specify custom logic for performing diffs on the relevant data set. By default a JSON.stringify approach is used. Different data sets might have more optimal implementations.

const watch = powersync
  .query({
    sql: /* sql */ `
      SELECT
        *
      FROM
        assets
    `,
    mapper: (raw) => {
      return {
        id: raw.id as string,
        make: raw.make as string
      };
    }
  })
  .differentialWatch({
    differentiator: {
      identify: (item) => item.id,
      compareBy: (item) => JSON.stringify(item)
    }
  });

Updates to query parameters can be performed in a single place, affecting all subscribers.

watch.updateSettings({
      query: new GetAllQuery({ sql: `SELECT * FROM assets OFFSET ? LIMIT 100`, parameters: [newOffset] })
    });

Reactivity

The existing watch method and Reactivity packages have been updated to use incremental queries with differentiation defined as an opt-in feature (defaults to no changes).

powersync.watch(
  'select * from assets',
  [],
  {
    onResult: () => {
      // This callback will be called whenever the data changes.
      console.log('Data updated');
    }
  },
  {
    comparator: {
      checkEquality: (current, previous) => {
        // This comparator will only report updates if the data changes.
        return JSON.stringify(current) === JSON.stringify(previous);
      }
    }
  }
);
/// React hooks
const { data, isLoading, isFetching } = useQuery(`SELECT * FROM cats WHERE breed = 'tabby'`, [], {
      differentiator: {
            identify: (item) => item.id,
            compareBy: (item) => JSON.stringify(item)
      }
  })

New hooks have also been added to use shared WatchedQuery instances.

const assetsQuery = powersync.query({
/// ....
}).watch();

/// In the component
export const MyComponent = () => {
   // In React one could import the `assetsQuery` or create a context provider for various queries
   const { data } = useWatchedQuerySubscription(assetsQuery)
}

The Vue and React hooks packages have been updated to remove duplicate implementations of common watched query logic. Historically these packages relied on manually implementing watched queries based off the onChange API in order to cater for exposing additional state and custom query executors. The new WatchedQuery APIs now support all the hook packages' requirements - this effectively reduces the heavy lifting in reactivity packages.

The React Supabase Todolist demo has been updated with some best practices for reducing re-renders using comparison based incrementally watched queries.

Differential Queries

These watched queries report the changes between result sets. The data member of a WatchedQuery's state is of the form

export interface WatchedQueryDifferential<RowType> {
  added: RowType[];
  all: RowType[];
  removed: RowType[];
  // This gives access to the current and previous values
  updated: WatchedQueryRowDifferential<RowType>[];
  unchanged: RowType[];
}

A common use case for this is processing newly created items as they are added. The YJS React Supabase Text Collab demo has been updated to take advantage of this feature. Document updates are watched via a differential incremental query. New updates are passed to YJS for consolidation as they are synced - the application code no longer needs to track which items were already passed to YJS.

Copy link

changeset-bot bot commented May 30, 2025

🦋 Changeset detected

Latest commit: c55dc01

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages
Name Type
@powersync/common Minor
@powersync/vue Minor
@powersync/react Minor
@powersync/web Minor
@powersync/node Patch
@powersync/op-sqlite Patch
@powersync/react-native Patch
@powersync/tanstack-react-query Patch
@powersync/diagnostics-app Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@stevensJourney stevensJourney requested a review from Copilot June 23, 2025 10:46
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces an incremental watched query system across the PowerSync platform, replacing the legacy table-change–based watches with comparison- and differential-based modes, and refactors the web, Vue, and React packages (including their tests and demos) to adopt the new API.

  • Adds incrementalWatch API to AbstractPowerSyncDatabase with COMPARISON and DIFFERENTIAL modes
  • Implements WatchedQuery, processors, and builders in @powersync/common
  • Updates Vue and React composables/hooks/tests to consume the new incremental watch API

Reviewed Changes

Copilot reviewed 86 out of 88 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
packages/web/tests/multiple_instances.test.ts DRYed stream instantiation; increased retry delay
packages/web/src/worker/sync/SharedSyncImplementation.ts Hardened _testUpdateAllStatuses, removed no-op code
packages/web/src/db/adapters/LockedAsyncDatabaseAdapter.ts Added abort‐on‐close logic and lock timeout support
packages/vue/src/composables/useWatchedQuery.ts New hook using incrementalWatch
packages/vue/src/composables/useSingleQuery.ts, useQuery.ts Refactored to share logic between single/watch hooks
packages/vue/tests/*.test.ts Switched to real PowerSyncDatabase and plugin setup
packages/react/src/hooks/watched/watch-utils.ts New query-compatibility helpers
packages/react/src/hooks/watched/* New useWatchedQuery, useSingleQuery, subscription
packages/react/src/index.ts Exports updated hooks
packages/common/src/client/watched/*.ts WatchedQuery, processors, comparators, builders
packages/common/src/utils/BaseObserver.ts, DataStream.ts Observer enhancements, error event support
AbstractPowerSyncDatabase incrementalWatch and onChange updates New entry point for incremental queries

@stevensJourney stevensJourney marked this pull request as ready for review June 23, 2025 12:01
Chriztiaan
Chriztiaan previously approved these changes Jun 24, 2025
Copy link
Contributor

@Chriztiaan Chriztiaan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy with the changeset, while massive. Well documented and structured.

const watchedQuery = this.incrementalWatch({ mode: IncrementalWatchMode.COMPARISON }).build<QueryResult | null>({
comparator,
watch: {
query: {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This formal query definition works, but a shorthand alternative makes for the query option that takes in the SQL/Params and always does an executeRead/getAll by default/implicitly would be nice.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do understand the sentiment behind this concern. I also thought about this and introduced the GetAllQuery helper for this reason.

It is currently possible to supply a GetAllQuery as the query option e.g.

const watch = powersync
      .incrementalWatch({
        mode: IncrementalWatchMode.COMPARISON
      })
      .build({
        watch: {
          query: new GetAllQuery({
            sql: 'SELECT * FROM assets',
            parameters: []
          }),
          placeholderData: []
        }
      });

Which is slightly more typing that just supplying an object of the form {sql: string, parameters?: any[]}, but IMO provides a better experience than the alternatives like:

Having query be an union type of {sql: string, parameters?: any[]} | WatchCompatibleQuery.

This gives bad autocomplete (IMO) if a dev was trying to create this option (it gives all possible options as a first suggestion).
image

Alternatively, a Discriminated Union type would also not be directly compatible with our current CompatibleQuery interface.

Copy link
Contributor

@rkistner rkistner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a big PR, so just giving some high-level comments to start with:

Internals & Hooks

I really like the refactoring to move more of the logic into the internals - it's nice that the hooks are now fairly thin wrappers around the internal methods.

incrementalWatch

The db.incrementalWatch API feels a little difficult to follow with the current structure. It feels like the API changes completely based on which mode you select, and should perhaps rather be two separate APIs?

Another option is to invert the API to start with the query part first, for example something like this:

db.createQuery(sql, params).differentialWatch(...)

This can avoid the explosion of new methods on the database itself, instead having the different methods on a new Query-type class. You could perhaps also have different methods for creating this from different sources, e.g. to create from a Kysely or Drizzle query.

Incremental comparison queries

I see the main benefit of these to avoid re-renders, to improve rendering performance. It would be good to see some benchmarks to validate that this is worth the additional complexity, that we're not just moving overhead from the rendering part to the query part.

I also wonder if we should unify comparitors and differentiators, always using the differentiator API (identify + compareBy instead of just compare). This way, you can do the re-rendering on an even more granular level. So if you have 1 row out of 1000 results that changed, you can keep the other 999 row objects from the previous results, avoiding further re-renders on those rows. Once again, we'd need some form of benchmarks to check that this is actually worth it.

* A hook to access and subscribe to the results of an existing {@link WatchedQuery} instance.
* @example
* export const ContentComponent = () => {
* const { data: lists } = useWatchedQuerySuspenseSubscription(listsQuery);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name is wrong in this example.

if (signal.aborted) {
return; // Abort if the signal is already aborted
}
console.log('done with query refresh');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should remove these log statements (including the one above)

@stevensJourney
Copy link
Collaborator Author

Update:

I've updated the implementation to guard the new watch methods behind the new query and customQuery methods on AbstractPowerSyncDatabase, following Ralf's recommendation. The PR description now includes usage examples.

Comparison based queries are now accessed via db.query().watch(). This watched query method does not perform a full diff on result sets; instead, it can exit early as soon as a change is detected. Use this method when you don't need to know the specifics of what changed and don't need to maintain previous object references in the result set. For cases where a full diff and object reference preservation are required, use db.query(...).differentialWatch().

The DifferentialWatchedQuery has been improved to better maintain previous array object references, further reducing unnecessary renders during incremental updates.

The WatchedQuery implementation now explicitly defines WatchedQueryState as readonly, making all state members and data items immutable. The Vue hooks are still in version 0; their APIs have been updated to also convey state as readonly. The React hooks package is already at a stable v1 release, so changing the API to report readonly state would be a breaking change. Backwards compatibility is maintained by conditionally applying readonly qualifiers if a differentiator parameter is supplied to the hook. Existing user code should not be affected.

An automated React profiling benchmark suite has been added in packages/react/tests/profile.test.tsx to measure the impact of differential queries on render performance. This test validates how using React memoization together with differential queries can significantly reduce render times. According to the results, incremental updates to the list widget render 60–80% faster compared to standard query methods. For details on the testing methodology, refer to the test file.
TLDR: The test starts with an initial amount of list items being rendered. We then incrementally add new items while measuring the render time of the entire list widget. Differential queries coupled with React memoization allow the incremental renders to only render the newly added item widgets, while the standard query methods re-render the entire widget. The graph below illustrates the render time for a list widget with various query and memoization methods.

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants