feat: add runtime validation to prevent custom getKey with joined queries #717

KyleAMathews · 2025-10-24T13:15:13Z

Implements fail-fast validation to catch the custom getKey + joins bug
at collection creation time instead of during sync.

Changes:

Added CustomGetKeyWithJoinError to provide clear error message
Added hasJoins() method that recursively checks query tree for joins
Validation runs in CollectionConfigBuilder constructor
Updated tests to verify error is thrown correctly
Added test for nested subquery join detection

The error message guides users to:

Remove custom getKey for joined queries
Use array methods like .toArray.find() instead of .get()

Prevents the CollectionOperationError and TransactionError that occurred
when sync tried to insert with composite keys "[key1,key2]" while the
collection expected simple keys from custom getKey.

Investigation of bug report where using custom getKey with joined queries causes CollectionOperationError and TransactionError. Root cause: Joined queries use composite keys like "[key1,key2]" internally, but custom getKey returns simple keys, creating a mismatch between the sync system and the collection. Solution: Do not use custom getKey with joined queries. The default getKey correctly uses the composite key from the internal WeakMap. - Added test cases demonstrating correct and incorrect usage - Created comprehensive investigation document with code references - Documented that joined results need composite keys for uniqueness 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

…ries Implements fail-fast validation to catch the custom getKey + joins bug at collection creation time instead of during sync. Changes: - Added CustomGetKeyWithJoinError to provide clear error message - Added hasJoins() method that recursively checks query tree for joins - Validation runs in CollectionConfigBuilder constructor - Updated tests to verify error is thrown correctly - Added test for nested subquery join detection The error message guides users to: - Remove custom getKey for joined queries - Use array methods like .toArray.find() instead of .get() Prevents the CollectionOperationError and TransactionError that occurred when sync tried to insert with composite keys "[key1,key2]" while the collection expected simple keys from custom getKey. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

changeset-bot · 2025-10-24T13:15:17Z

🦋 Changeset detected

Latest commit: 28e8b96

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@tanstack/db	Patch
@tanstack/angular-db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/react-db	Patch
@tanstack/rxdb-db-collection	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch
todos	Patch
@tanstack/db-example-react-todo	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

pkg-pr-new · 2025-10-24T13:16:55Z

More templates

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@717

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@717

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@717

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@717

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@717

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@717

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@717

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@717

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@717

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@717

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@717

commit: 28e8b96

github-actions · 2025-10-24T13:18:21Z

Size Change: +502 B (+0.59%)

Total Size: 85 kB

Filename	Size	Change
`./packages/db/dist/esm/collection/sync.js`	2.26 kB	+54 B (+2.45%)
`./packages/db/dist/esm/errors.js`	3.75 kB	+265 B (+7.61%)	🔍
`./packages/db/dist/esm/query/live/collection-config-builder.js`	5.73 kB	+183 B (+3.3%)

ℹ️ View Unchanged

Filename	Size
`./packages/db/dist/esm/collection/change-events.js`	1.63 kB
`./packages/db/dist/esm/collection/changes.js`	1.01 kB
`./packages/db/dist/esm/collection/events.js`	413 B
`./packages/db/dist/esm/collection/index.js`	3.23 kB
`./packages/db/dist/esm/collection/indexes.js`	1.16 kB
`./packages/db/dist/esm/collection/lifecycle.js`	1.8 kB
`./packages/db/dist/esm/collection/mutations.js`	2.52 kB
`./packages/db/dist/esm/collection/state.js`	3.8 kB
`./packages/db/dist/esm/collection/subscription.js`	2.2 kB
`./packages/db/dist/esm/deferred.js`	230 B
`./packages/db/dist/esm/event-emitter.js`	798 B
`./packages/db/dist/esm/index.js`	1.62 kB
`./packages/db/dist/esm/indexes/auto-index.js`	857 B
`./packages/db/dist/esm/indexes/base-index.js`	835 B
`./packages/db/dist/esm/indexes/btree-index.js`	2 kB
`./packages/db/dist/esm/indexes/lazy-index.js`	1.21 kB
`./packages/db/dist/esm/indexes/reverse-index.js`	577 B
`./packages/db/dist/esm/local-only.js`	967 B
`./packages/db/dist/esm/local-storage.js`	2.42 kB
`./packages/db/dist/esm/optimistic-action.js`	294 B
`./packages/db/dist/esm/proxy.js`	3.86 kB
`./packages/db/dist/esm/query/builder/functions.js`	615 B
`./packages/db/dist/esm/query/builder/index.js`	4.04 kB
`./packages/db/dist/esm/query/builder/ref-proxy.js`	938 B
`./packages/db/dist/esm/query/compiler/evaluators.js`	1.55 kB
`./packages/db/dist/esm/query/compiler/expressions.js`	760 B
`./packages/db/dist/esm/query/compiler/group-by.js`	2.04 kB
`./packages/db/dist/esm/query/compiler/index.js`	2.21 kB
`./packages/db/dist/esm/query/compiler/joins.js`	2.65 kB
`./packages/db/dist/esm/query/compiler/order-by.js`	1.43 kB
`./packages/db/dist/esm/query/compiler/select.js`	1.28 kB
`./packages/db/dist/esm/query/ir.js`	785 B
`./packages/db/dist/esm/query/live-query-collection.js`	404 B
`./packages/db/dist/esm/query/live/collection-registry.js`	233 B
`./packages/db/dist/esm/query/live/collection-subscriber.js`	2.11 kB
`./packages/db/dist/esm/query/optimizer.js`	3.33 kB
`./packages/db/dist/esm/scheduler.js`	1.29 kB
`./packages/db/dist/esm/SortedMap.js`	1.24 kB
`./packages/db/dist/esm/transactions.js`	3.05 kB
`./packages/db/dist/esm/utils.js`	1.01 kB
`./packages/db/dist/esm/utils/browser-polyfills.js`	365 B
`./packages/db/dist/esm/utils/btree.js`	6.01 kB
`./packages/db/dist/esm/utils/comparison.js`	754 B
`./packages/db/dist/esm/utils/index-optimization.js`	1.73 kB

_{compressed-size-action::db-package-size}

- Added changeset for custom getKey validation fix - Removed investigation document (not needed in PR) - Simplified and focused tests on validation behavior - Ran prettier to format code Tests now clearly demonstrate: 1. Joins work without custom getKey 2. Error thrown when custom getKey used with joins 3. Nested subquery joins are detected 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

github-actions · 2025-10-24T13:23:46Z

Size Change: 0 B

Total Size: 2.89 kB

ℹ️ View Unchanged

Filename	Size
`./packages/react-db/dist/esm/index.js`	168 B
`./packages/react-db/dist/esm/useLiveInfiniteQuery.js`	1.41 kB
`./packages/react-db/dist/esm/useLiveQuery.js`	1.31 kB

_{compressed-size-action::react-db-package-size}

Fixed TypeScript errors and test logic: - Added .select() to create proper result types for joined queries - Fixed getKey to access the selected properties (baseId instead of id) - Fixed nested subquery test to use actual live query collection instead of function - Properly tests that validation detects joins in nested subqueries All tests now properly validate the runtime error checking. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Removed the test for detecting joins in nested live query collections as it requires more complex detection logic that's not implemented yet. The edge case where you reference a live query collection (which internally has joins) would require checking if the source collection is a live query and recursively inspecting its query definition. The two core test cases still validate: 1. Joins work correctly without custom getKey 2. Error is thrown when custom getKey is used with direct joins All tests now pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

samwillis · 2025-10-24T18:06:49Z

@KyleAMathews could you fill me in on your thinking here? The intention is that a user can set a getKey on a joined query to specify a natural key to use rather than the generated on. We should fix the underlying issue not prevent this option as it was intended to be possible.

samwillis · 2025-10-24T18:07:42Z

It may be that there is a bug somewhere where it is not using the getKey when materialising.

KyleAMathews · 2025-10-24T18:32:51Z

@samwillis this was the original bug report https://discord.com/channels/933657521581858818/933657523049885698/1431144066099183656:

client.ts:1032 Uncaught CollectionOperationError: Cannot insert document with key "693892a1-4d2e-4f8d-9c05-9e9c5c667ca1" from sync because it already exists in the collection "live-query-1
 
client.ts:1032 Uncaught TransactionError: The pending sync transaction is already committed, you can't commit it again.

Which seemed like a duplicate key problem caused by their custom getKey that was just returning the id from one of the joined collections.

samwillis · 2025-10-31T10:58:28Z

@KyleAMathews This is a slightly tough one to solve in a generic way - an example of a query where a user is likely to want a custom getKey is joins users to a comments collection. They know with certainty there is going to be a single row per comment and so a getKey that sets it to the comments ID is ideal, they don't really want to key that collection by [commentId,userId]. I don't think we can particularly easily infer these type of situations from the query though.

Maybe the solution is that a LiveQueryCollection overloads the CollectionOperationError error for inserts and producers a better error message when there is a custom getKey + join. Have it indicate that their getKey unexpectedly produced a duplicate key and they likely need to generate a composite key or refactor the join to do create duplicates.

…g-011CUS4ez8csReqMepDi5gRm

@samwillis

…stead of blocking Implemented Sam's better approach: instead of blocking all custom getKey with joins upfront, we now allow it and provide enhanced error messages when duplicate keys actually occur during sync. Changes: - Removed upfront validation that blocked custom getKey + joins - Enhanced DuplicateKeySyncError with context-aware messaging - Added metadata (_hasCustomGetKey, _hasJoins) to collection utils - Updated sync.ts to pass context when throwing duplicate key errors - Removed unused CustomGetKeyWithJoinError class - Updated tests to show custom getKey works with joins (1:1 cases) - Updated changeset to reflect the new approach Benefits: - Allows valid 1:1 join cases with custom getKey - Only errors when actual duplicates occur (fail-fast on real problems) - Provides helpful guidance with composite key examples - More flexible for users who know their data structure Example enhanced error: "Cannot insert document with key "user1" from sync because it already exists. This collection uses a custom getKey with joined queries. Joined queries can produce multiple rows with the same key when relationships are not 1:1. Consider: (1) using a composite key (e.g., `${item.key1}-${item.key2}`), (2) ensuring your join produces unique rows per key, or (3) removing the custom getKey to use the default composite key behavior." Credit: Suggested by @samwillis 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Removed references to intermediate solutions and rewrote from the perspective of main branch - what problem this solves and what it adds. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Changed _hasCustomGetKey and _hasJoins from boolean properties to functions that return booleans. This fixes the TypeScript build error since UtilsRecord requires all properties to be functions. - Updated collection-config-builder.ts to use arrow functions - Updated sync.ts to call the functions with () - Tests still pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

KyleAMathews · 2025-10-31T15:17:53Z

@samwillis ok ready for another review

claude added 2 commits October 24, 2025 13:05

claude added 2 commits October 24, 2025 13:25

KyleAMathews requested a review from samwillis October 24, 2025 13:53

claude added 4 commits October 31, 2025 13:36

Merge remote-tracking branch 'origin/main' into claude/investigate-bu…

5d0b634

…g-011CUS4ez8csReqMepDi5gRm

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add runtime validation to prevent custom getKey with joined queries #717

feat: add runtime validation to prevent custom getKey with joined queries #717

KyleAMathews commented Oct 24, 2025

Uh oh!

changeset-bot bot commented Oct 24, 2025 •

edited

Loading

Uh oh!

pkg-pr-new bot commented Oct 24, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 24, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 24, 2025 •

edited

Loading

Uh oh!

samwillis commented Oct 24, 2025

Uh oh!

samwillis commented Oct 24, 2025

Uh oh!

KyleAMathews commented Oct 24, 2025

Uh oh!

samwillis commented Oct 31, 2025

Uh oh!

KyleAMathews commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat: add runtime validation to prevent custom getKey with joined queries #717

Are you sure you want to change the base?

feat: add runtime validation to prevent custom getKey with joined queries #717

Conversation

KyleAMathews commented Oct 24, 2025

Uh oh!

changeset-bot bot commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

pkg-pr-new bot commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samwillis commented Oct 24, 2025

Uh oh!

samwillis commented Oct 24, 2025

Uh oh!

KyleAMathews commented Oct 24, 2025

Uh oh!

samwillis commented Oct 31, 2025

Uh oh!

KyleAMathews commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

changeset-bot bot commented Oct 24, 2025 •

edited

Loading

pkg-pr-new bot commented Oct 24, 2025 •

edited

Loading

github-actions bot commented Oct 24, 2025 •

edited

Loading

github-actions bot commented Oct 24, 2025 •

edited

Loading