Skip to content

Commit b2eb96d

Browse files
KyleAMathewsclaude
andauthored
Improve schema documentation (#741)
* Add detailed response explaining TanStack DB schema types to PowerSync team This document clarifies the TInput/TOutput architecture and explains how PowerSync can support arbitrary type transformations (like Date objects) by handling serialization in their integration layer rather than constraining TOutput to match SQLite types. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * Add comprehensive schema documentation proposal This proposal addresses the lack of documentation around TInput/TOutput schema types and transformations. It includes: - Complete content outline for new schemas.md guide - Data flow diagrams and examples - Guidance for both app developers and integration authors - Common patterns for Date handling, defaults, and type conversions - Updates to existing docs (overview, mutations, collection-options-creator) The proposal directly addresses confusion like what the PowerSync team experienced regarding how to handle type transformations and serialization in integrations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * Add refined schema documentation plan based on deep investigation After investigating all existing docs (overview, mutations, error-handling, live-queries, collection-options-creator) and examples, created a refined plan that addresses: KEY FINDING: Two distinct type conversion mechanisms 1. Integration-level parsing (storage format ↔ in-memory format) 2. Schema validation/transformation (TInput → TOutput for mutations) The plan includes: - Analysis of what's currently documented (and gaps) - Comprehensive schemas.md guide structure (11 sections) - Specific updates to 5 existing docs with exact content - Separate guidance for app developers vs integration authors - Clear distinction between integration parsing and schema validation - Complete working examples and best practices - Implementation order and success criteria This directly addresses the PowerSync confusion about TInput/TOutput and provides clear guidance for both audiences. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * Add comprehensive schemas guide for TanStack DB New guide (docs/guides/schemas.md) covering: - Introduction with validation-first example - Core concepts: TInput vs TOutput with data flow diagram - Validation patterns (types, strings, numbers, enums, arrays, custom) - Transformation patterns (Date conversion, JSON, computed fields) - Default values (literals, functions, complex) - Handling updates with union types pattern - Error handling with SchemaValidationError - Best practices (with performance callout) - Two complete working examples (Todo app, E-commerce) - Brief integration authors section linking to collection-options-creator - Related topics links This addresses the documentation gap identified in the PowerSync question about TInput/TOutput and provides clear guidance for both app developers and integration authors on schema validation and type transformations. ~850 lines, 12 sections, 30+ code examples 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * refactor: revise schemas.md based on feedback Address review feedback to make the guide more focused and practical: - Add clarification that schemas only validate client changes, not server data - Remove "Handling Sync Validation Errors" section - Fix QueryFn example to show manual parsing is required for API responses - Rename "Handling Updates" to "Handling Timestamps" with better focus on common patterns - Remove "Safe Parsing (Zod)" section - Remove "When to Use Schemas" from Best Practices - Remove "Schema Organization" from Best Practices - Replace lengthy "For Integration Authors" section with brief link to collection-options-creator.md 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * docs: clarify schema validation and improve queryFn example - Explicitly mention schemas catch invalid data from optimistic mutations - Show reusing schema with .parse() in queryFn to transform API responses - Remove The Data Flow diagram section (had errors and wasn't useful) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * docs: update all docs with schema information Updates across documentation to explain schemas and type transformations: **overview.md:** - Expand collection schemas section with comprehensive example - Add list of what schemas do (validation, transformations, defaults, type safety) - Link to new schemas guide **mutations.md:** - Add "Schema Validation in Mutation Handlers" section - Explain that handlers receive TOutput (transformed data) - Show serialization pattern for backends **error-handling.md:** - Add "When schema validation occurs" section - Clarify schemas only validate client mutations, not sync data - Link to schemas guide **collection-options-creator.md:** - Add "Schemas and Type Transformations" section - Explain three approaches: parse/serialize helpers, user handles, automatic serialization - Show examples from TrailBase and Query Collection - Document design principles for integration authors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * Remove docs * docs: clarify TInput must be superset of TOutput requirement Addresses PR review feedback from samwillis about the critical design principle that TInput must be a superset of TOutput when using transformations. Key improvements: - Add prominent "Critical Design Principle" section explaining why TInput must accept all TOutput values - Clarify that union types are REQUIRED (not optional) for transformations - Add clear ❌/✅ examples showing what breaks and why - Explain the draft parameter typing issue in collection.update() - Strengthen language in Best Practices from "should" to "must" This makes it clear that when schemas transform type A to type B, you must use z.union([A, B]) to ensure updates work correctly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> --------- Co-authored-by: Claude <[email protected]>
1 parent 3ee1d7e commit b2eb96d

File tree

5 files changed

+1326
-5
lines changed

5 files changed

+1326
-5
lines changed

docs/guides/collection-options-creator.md

Lines changed: 118 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -219,7 +219,124 @@ parse: {
219219
}
220220
```
221221

222-
### 5. Mutation Handler Patterns
222+
### 5. Schemas and Type Transformations
223+
224+
When building a custom collection, you need to decide how to handle the relationship between your backend's storage format and the client-side types users work with in their collections.
225+
226+
#### Two Separate Concerns
227+
228+
**Backend Format** - The types your storage layer uses (SQLite, Postgres, Firebase, etc.)
229+
- Examples: Unix timestamps, ISO strings, JSON strings, PostGIS geometries
230+
231+
**Client Format** - The types users work with in their TanStack DB collections
232+
- Examples: Date objects, parsed JSON, GeoJSON
233+
234+
Schemas in TanStack DB define the **client format** (TInput/TOutput for mutations). How you bridge between backend and client format depends on your integration design.
235+
236+
#### Approach 1: Integration Provides Parse/Serialize Helpers
237+
238+
For backends with specific storage formats, provide `parse`/`serialize` options that users configure:
239+
240+
```typescript
241+
// TrailBase example: User specifies field conversions
242+
export function trailbaseCollectionOptions(config) {
243+
return {
244+
parse: config.parse, // User provides field conversions
245+
serialize: config.serialize,
246+
247+
onInsert: async ({ transaction }) => {
248+
const serialized = transaction.mutations.map(m =>
249+
serializeFields(m.modified, config.serialize)
250+
)
251+
await config.recordApi.createBulk(serialized)
252+
}
253+
}
254+
}
255+
256+
// User explicitly configures conversions
257+
const collection = createCollection(
258+
trailbaseCollectionOptions({
259+
schema: todoSchema,
260+
parse: {
261+
created_at: (ts: number) => new Date(ts * 1000) // Unix → Date
262+
},
263+
serialize: {
264+
created_at: (date: Date) => Math.floor(date.valueOf() / 1000) // Date → Unix
265+
}
266+
})
267+
)
268+
```
269+
270+
**Benefits:** Explicit control over type conversions. Integration handles applying them consistently.
271+
272+
#### Approach 2: User Handles Everything in QueryFn/Handlers
273+
274+
For simple APIs or when users want full control, they handle parsing/serialization themselves:
275+
276+
```typescript
277+
// Query Collection: User handles all transformations
278+
const collection = createCollection(
279+
queryCollectionOptions({
280+
schema: todoSchema,
281+
queryFn: async () => {
282+
const response = await fetch('/api/todos')
283+
const todos = await response.json()
284+
// User manually parses to match their schema's TOutput
285+
return todos.map(todo => ({
286+
...todo,
287+
created_at: new Date(todo.created_at) // ISO string → Date
288+
}))
289+
},
290+
onInsert: async ({ transaction }) => {
291+
// User manually serializes for their backend
292+
await fetch('/api/todos', {
293+
method: 'POST',
294+
body: JSON.stringify({
295+
...transaction.mutations[0].modified,
296+
created_at: transaction.mutations[0].modified.created_at.toISOString() // Date → ISO string
297+
})
298+
})
299+
}
300+
})
301+
)
302+
```
303+
304+
**Benefits:** Maximum flexibility, no abstraction overhead. Users see exactly what's happening.
305+
306+
#### Approach 3: Automatic Serialization in Handlers
307+
308+
If your backend has well-defined types, you can automatically serialize in mutation handlers:
309+
310+
```typescript
311+
export function myCollectionOptions(config) {
312+
return {
313+
onInsert: async ({ transaction }) => {
314+
// Automatically serialize known types for your backend
315+
const serialized = transaction.mutations.map(m => ({
316+
...m.modified,
317+
// Date objects → Unix timestamps for your backend
318+
created_at: m.modified.created_at instanceof Date
319+
? Math.floor(m.modified.created_at.valueOf() / 1000)
320+
: m.modified.created_at
321+
}))
322+
await backend.insert(serialized)
323+
}
324+
}
325+
}
326+
```
327+
328+
**Benefits:** Least configuration for users. Integration handles backend format automatically.
329+
330+
#### Key Design Principles
331+
332+
1. **Schemas validate client mutations only** - They don't affect how backend data is parsed during sync
333+
2. **TOutput is the application-facing type** - This is what users work with in their app
334+
3. **Choose your approach based on backend constraints** - Fixed types → automatic serialization; varying types → user configuration
335+
4. **Document your backend format clearly** - Explain what types your storage uses and how to handle them
336+
337+
For more on schemas from a user perspective, see the [Schemas guide](./schemas.md).
338+
339+
### 6. Mutation Handler Patterns
223340

224341
There are two distinct patterns for handling mutations in collection options creators:
225342

docs/guides/error-handling.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,32 @@ The error includes:
4545
- `issues`: Array of validation issues with messages and paths
4646
- `message`: A formatted error message listing all issues
4747

48+
**When schema validation occurs:**
49+
50+
Schema validation happens only for **client mutations** - when you explicitly insert or update data:
51+
52+
1. **During inserts** - When `collection.insert()` is called
53+
2. **During updates** - When `collection.update()` is called
54+
55+
Schemas do **not** validate data coming from your server or sync layer. That data is assumed to already be valid.
56+
57+
```typescript
58+
const schema = z.object({
59+
id: z.string(),
60+
created_at: z.string().transform(val => new Date(val))
61+
// TInput: string, TOutput: Date
62+
})
63+
64+
// Validation happens here ✓
65+
collection.insert({
66+
id: "1",
67+
created_at: "2024-01-01" // TInput: string
68+
})
69+
// If successful, stores: { created_at: Date } // TOutput: Date
70+
```
71+
72+
For more details on schema validation and type transformations, see the [Schemas guide](./schemas.md).
73+
4874
## Query Collection Error Tracking
4975

5076
Query collections provide enhanced error tracking utilities through the `utils` object. These methods expose error state information and provide recovery mechanisms for failed queries:

docs/guides/mutations.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -486,6 +486,49 @@ const todoCollection = createCollection({
486486
})
487487
```
488488

489+
### Schema Validation in Mutation Handlers
490+
491+
When a schema is configured for a collection, TanStack DB automatically validates and transforms data during mutations. The mutation handlers receive the **transformed data** (TOutput), not the raw input.
492+
493+
```typescript
494+
const todoSchema = z.object({
495+
id: z.string(),
496+
text: z.string(),
497+
created_at: z.string().transform(val => new Date(val)) // TInput: string, TOutput: Date
498+
})
499+
500+
const collection = createCollection({
501+
schema: todoSchema,
502+
onInsert: async ({ transaction }) => {
503+
const item = transaction.mutations[0].modified
504+
505+
// item.created_at is already a Date object (TOutput)
506+
console.log(item.created_at instanceof Date) // true
507+
508+
// If your API needs a string, serialize it
509+
await api.todos.create({
510+
...item,
511+
created_at: item.created_at.toISOString() // Date → string
512+
})
513+
}
514+
})
515+
516+
// User provides string (TInput)
517+
collection.insert({
518+
id: "1",
519+
text: "Task",
520+
created_at: "2024-01-01T00:00:00Z"
521+
})
522+
```
523+
524+
**Key points:**
525+
- Schema validation happens **before** mutation handlers are called
526+
- Handlers receive **TOutput** (transformed data)
527+
- If your backend needs a different format, serialize in the handler
528+
- Schema validation errors throw `SchemaValidationError` before handlers run
529+
530+
For comprehensive documentation on schema validation and transformations, see the [Schemas guide](./schemas.md).
531+
489532
## Creating Custom Actions
490533

491534
For more complex mutation patterns, use `createOptimisticAction` to create custom actions with full control over the mutation lifecycle.

0 commit comments

Comments
 (0)