feat: custom serialization #7223

wmertens · 2025-01-02T12:26:30Z

This PR adds symbols to mark objects as (no/yes)serializable via symbols instead of via a Set, adds a symbol prop that will get called on serialization for custom serialization, and provides a new Signal type that lazily manages a non-serializable value.

Note that this is basically the same as useComputed$, except that it is invalidated during SSR so that when the value gets read on the client it will always run the compute function.

Also note that useComputed$ now passes the previous value to the compute function, to keep the implementation compact. This seems like it might be useful, should we document this?

Everything works, looking for comments on implementation, naming etc.

TODO

update eslint to accept serialized signals and the new serializer symbols
check crash that was reported

changeset-bot · 2025-01-02T12:26:33Z

🦋 Changeset detected

Latest commit: 54acbc0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages

Name	Type
@qwik.dev/core	Major
eslint-plugin-qwik	Major
@qwik.dev/react	Major
@qwik.dev/router	Major
create-qwik	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Varixo · 2025-01-02T13:43:24Z

Looks great, but what about deserialization those objects?

wmertens · 2025-01-02T20:03:33Z

Looks great, but what about deserialization those objects?

That's what the createSerialized$ does

thejackshelton · 2025-01-02T22:33:56Z

Does this mean we might be able to serialize functions from other libraries, like vanilla js?

wmertens · 2025-01-03T07:27:47Z

@Varixo @thejackshelton gaah I had forgotten to push the actual commit 😅

wmertens · 2025-01-03T07:29:55Z

Does this mean we might be able to serialize functions from other libraries, like vanilla js?

Yes exactly. You need to add a symbol prop that serializes the value and then provide a deserializer that will get called lazily.

pkg-pr-new · 2025-01-03T08:10:03Z

Open in Stackblitz

npm i https://pkg.pr.new/QwikDev/qwik/@qwik.dev/core@7223

npm i https://pkg.pr.new/QwikDev/qwik/@qwik.dev/router@7223

npm i https://pkg.pr.new/QwikDev/qwik/eslint-plugin-qwik@7223

npm i https://pkg.pr.new/QwikDev/qwik/create-qwik@7223

commit: 54acbc0

github-actions · 2025-01-03T08:10:21Z

built with Refined Cloudflare Pages Action

⚡ Cloudflare Pages Deployment

Name	Status	Preview	Last Commit
qwik-docs	✅ Ready (View Log)	Visit Preview	`54acbc0`

packages/qwik/src/core/use/use-serializer.ts

packages/qwik/src/core/tests/use-computed.spec.tsx

packages/qwik/src/core/shared/shared-serialization.ts

Varixo · 2025-02-07T16:13:34Z

packages/qwik/src/core/shared/shared-serialization.ts

+            : obj instanceof ComputedSignal &&
+                !(obj instanceof SerializerSignal) &&
+                (obj.$invalid$ || fastSkipSerialize(obj))
              ? NEEDS_COMPUTATION
              : obj.$untrackedValue$;
        if (v !== NEEDS_COMPUTATION) {
-          discoveredValues.push(v);
+          if (obj instanceof SerializerSignal) {
+            promises.push(
+              (obj.$computeQrl$ as any as QRLInternal<SerializerArg<any, any>>)
+                .resolve()
+                .then((arg) => {
+                  let data;
+                  if ((arg as any).serialize) {
+                    data = (arg as any).serialize(v);
+                  }
+                  if (data === undefined) {
+                    data = NEEDS_COMPUTATION;
+                  }
+                  serializationResults.set(obj, data);
+                  discoveredValues.push(data);
+                })
+            );
+          } else {
+            discoveredValues.push(v);
+          }


I wonder if we could organise it somehow? It is very complicated right now.
What about doing just

if (obj instanceof WrappedSignal) { } else if (obj instanceof ComputedSignal) { } else if ...

thejackshelton

Very thorough PR. Awesome work ⚡

packages/docs/src/routes/api/qwik/index.md

packages/qwik/src/core/shared/shared-serialization.ts

thejackshelton · 2025-02-07T17:09:04Z

packages/qwik/src/core/signal/signal.ts

+       *
+       * This function must not return a promise.
+       */
+      deserialize: (data: S | undefined, previous: T | undefined) => T;


So on first render, the previous param is -> initial data / inital (3rd argument param of useSerializer$)?

on first run, the data is initial|undef. After serialization, data is what was serialized.

If the function uses reactive scope, it will rerun and then current will be defined and data will be undefined.

thejackshelton · 2025-02-07T17:10:46Z

packages/qwik/src/core/signal/signal.ts

+       *
+       * If you do not provide it, the object will be serialized as `undefined`.
+       */
+      serialize?: (customObject: T) => S | Promise<S>;


What's an example of when you may need the deserialize function but not the serialize function?

When you're just constructing an object from scope data and you don't have any state to serialize.

thejackshelton-kunaico

wrong account 😓

ianlet · 2025-02-13T09:46:14Z

I tried the changes locally and I'm having the following issue when using props as initial data:

Plus a linting issue when using the SerializerSignal in a useTask$.

ESLint: 
When referencing "mapbox" inside a different scope (useTask$), Qwik needs to serialize the value, however "mapbox. force" is a function, which is not serializable. Check out https:// qwik. dev/ docs/ advanced/ dollar/ for more details.
(qwik/ valid-lexical-scope)

Link to MRE: https://github.com/ianlet/qwik-custom-serder-scope-issues/blob/main/src/components/qwik-mapbox.tsx

Edit: actually we won't be able to useSerializer for mapbox as it needs a window/document. Could have been nice though to render a map without having to wake up Qwik. In any case, it was an interesting exercise to try this new feature on a more complex object that we don't control (external lib).

ianlet · 2025-02-13T09:55:51Z

Also, after playing with it a little bit, I think the deserialize function should not handle both the actual deserialization and the update when tracked values change. It's conceptually 2 different things and will require 2 different implementations anyway for developers who will use it. And the fact that we had to document 2 examples on how to use it (without tracking and with tracking) reinforces this idea.

So, IMO it should be 2 different callbacks as it is 2 different use cases, triggered after 2 different events.

wmertens · 2025-02-13T22:11:09Z

I addressed all the comments I think.

wmertens · 2025-02-13T22:27:04Z

@ianlet BTW even when only using mapbox on the client, this is still useful. Just make the deserialize isBrowser ? makeMapbox() : undefined and it will just work on the client when you need it.

ianlet · 2025-02-14T08:30:32Z

@ianlet BTW even when only using mapbox on the client, this is still useful. Just make the deserialize isBrowser ? makeMapbox() : undefined and it will just work on the client when you need it.

I tried to use the new syntax with update and I changed it to apply your isServer suggestion but if I do so, the deserialize function is never called on the client so the value never gets initialized.

Also, having to handle the undefined everywhere in that case makes it a bit heavy, but it's a detail.

    const mapbox = useSerializer$<
      mapboxgl.Map | undefined,
      SerializationState | undefined
    >(() => ({
      initial: {
        token: token.value,
        mapStyle: mapStyle.value,
        center: center.value,
      },
      serialize: (map: mapboxgl.Map | undefined) => {
        if (!map) return;

        return {
          token,
          mapStyle: map.getStyle() || undefined,
          center: map.getCenter().toArray(),
        };
      },
      deserialize: (state: SerializationState | undefined) => {
        if (isServer) return;

        const s = state || { token: token.value, mapStyle: mapStyle.value, center: center.value };

        return new mapboxgl.Map({
          container: ref.value!,
          style: s.mapStyle,
          center: s.center,
        });
      },
      update: (map: mapboxgl.Map | undefined) => {
        if (!map) return;

        map.setStyle(mapStyle.value);
        map.setCenter(center.value);
      },
    }));

Note: ESLint is still not happy about tracking the SerializerSignal:

    useTask$(({ track }) => {
      const map = track(mapbox);

      map?.on("load", () => console.log("MAP LOADED"));
    });

5:28:38 PM [vite] warning: When referencing "mapbox" inside a different scope (useTask$), Qwik needs to serialize the value, however "mapbox.force" is a function, which is not serializable.
Check out https://qwik.dev/docs/advanced/dollar/ for more details.
  Plugin: vite-plugin-qwik
  File: qwik-custom-serder/src/components/qwik-mapbox.tsx:65:25
  63 |
  64 |      useTask$(({ track }) => {
  65 |        const map = track(mapbox);
     |                           ^
  66 |
  67 |        map?.on("load", () => console.log("MAP LOADED"));

ianlet · 2025-02-14T09:18:00Z

I played more with it and if you change a value that is tracked more than once, the serialized value is only updated the first time: Playground #1

So I tried to change it to directly use the signal in the deserialize function (feels weird to me though), and this time it crashes if updated more than once: Playground #2

Uncaught (in promise) TypeError: p0.value is undefined
    qFuncs_za6h4ekne3o (index):1
    untrackedValue signal.js:405
    invokeApply use-core.js:62
    invoke use-core.js:54
    trackSignal use-core.js:130
    $computeIfNeeded$ signal.js:405
    returnValue scheduler.js:349
    retryOnPromise promises.js:88
    executeChore scheduler.js:348
    drainUpTo scheduler.js:212
    schedule scheduler.js:179
    queueQRL queue-qrl.js:21
    dispatch (index):140
    processDocumentEvent (index):162
    addEventListener (index):194
    processEventOrNode (index):202
    processEventOrNode (index):202
    <anonymous> (index):225

PR Overview

This PR introduces custom serialization support by adding two new symbols (NoSerializeSymbol and SerializerSymbol) to control object serialization and by providing a new SerializerSignal type that enables custom serialization logic. Key changes include updates to the serialization and signal modules, added unit tests demonstrating custom serialization, and comprehensive documentation updates.

Reviewed Changes

File	Description
packages/qwik/src/core/shared/shared-serialization.unit.ts	Adds unit tests for NoSerializeSymbol and SerializerSymbol, as well as SerializerSignal behavior.
packages/qwik/src/core/signal/signal.ts	Introduces type alias and minor refactoring for computed and wrapped signals, including read-only value enforcement.
packages/qwik/src/core/shared/shared-serialization.ts	Updates to support SerializerSignal in both allocation and serialization flows.
packages/qwik/src/core/signal/signal.public.ts & signal-api.ts	Exposes new createSerializer$ and createSerializerQrl APIs with improved type definitions and internal casting.
Documentation and changeset files	Updates to document the two new symbols and the custom serialization API.

Copilot reviewed 32 out of 32 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (2)

packages/qwik/src/core/shared/shared-serialization.ts:312

[nitpick] Consider expanding this comment to further clarify why SerializerSignal is always marked as invalid, ensuring future maintainers understand the rationale behind recreating the custom object during deserialization.

        // The serialized signal is always invalid so it can recreate the custom object

packages/qwik/src/core/signal/signal-api.ts:32

[nitpick] Consider refining the type casting here to reduce the double 'any' cast and make the conversion to QRLInternal<SerializerArg<T, S>> more type-safe.

return new SerializerSignal<T, S>(null, arg as any as QRLInternal<SerializerArg<T, S>>);

mhevery

Very excited about this change. Left few clarifing questions.

.changeset/nasty-planes-jam.md

packages/qwik/src/core/shared/serialization.md

packages/eslint-plugin-qwik/tests/valid-lexical-scope/valid-no-serialize-symbol.tsx

packages/qwik/src/core/shared/shared-serialization.ts

packages/qwik/src/core/api.md

packages/qwik/src/optimizer/src/plugins/rollup.ts

packages/qwik/src/optimizer/src/qwik-binding-map.ts

Co-authored-by: Varixo <[email protected]> Co-authored-by: Miško Hevery <[email protected]>

Varixo

LGTM

packages/qwik/src/core/signal/signal-api.ts

wmertens · 2025-03-18T07:15:46Z

@ianlet Thank you for your playgrounds, the problem is that you didn't use foo.value in the deserializer, and so there's no subscription being created.

I added an eslint rule (yey vibe coding) that warns about this.

all points addressed

wmertens force-pushed the custom-serialize branch from aca1de8 to 47942f4 Compare January 3, 2025 07:27

wmertens force-pushed the custom-serialize branch from 47942f4 to 9009520 Compare January 3, 2025 07:56

Varixo reviewed Jan 3, 2025

View reviewed changes

packages/qwik/src/core/use/use-serializer.ts Outdated Show resolved Hide resolved

packages/qwik/src/core/tests/use-computed.spec.tsx Outdated Show resolved Hide resolved

packages/qwik/src/core/shared/shared-serialization.ts Show resolved Hide resolved

wmertens force-pushed the custom-serialize branch from 9009520 to 103e3c1 Compare January 4, 2025 10:44

wmertens mentioned this pull request Jan 7, 2025

RFC: useSerialized$/createSerialized$ + SerializeSymbol QwikDev/qwik-evolution#204

Closed

wmertens force-pushed the custom-serialize branch 2 times, most recently from 165ac6d to 804f01d Compare January 27, 2025 23:32

wmertens force-pushed the custom-serialize branch from 804f01d to 9070c30 Compare February 6, 2025 15:50

wmertens marked this pull request as ready for review February 6, 2025 15:50

wmertens requested review from a team as code owners February 6, 2025 15:50

Varixo reviewed Feb 7, 2025

View reviewed changes

thejackshelton approved these changes Feb 7, 2025

View reviewed changes

thejackshelton-kunaico suggested changes Feb 7, 2025

View reviewed changes

thejackshelton-kunaico approved these changes Feb 7, 2025

View reviewed changes

wmertens force-pushed the custom-serialize branch from 9070c30 to d67bb36 Compare February 13, 2025 22:05

wmertens force-pushed the custom-serialize branch from d67bb36 to 5126138 Compare February 14, 2025 07:46

Varixo requested a review from Copilot March 2, 2025 20:45

Copilot AI reviewed Mar 2, 2025

View reviewed changes

mhevery previously requested changes Mar 5, 2025

View reviewed changes

Varixo reviewed Mar 9, 2025

View reviewed changes

packages/qwik/src/optimizer/src/qwik-binding-map.ts Outdated Show resolved Hide resolved

wmertens force-pushed the custom-serialize branch from d76b001 to d7f7793 Compare March 17, 2025 16:27

wmertens and others added 3 commits March 17, 2025 19:15

fix(core): correct diagnostics type

e04df43

chore(core signals): cleanup

ba1bf52

feat(useSerializer$): custom serialization

4d056d4

Co-authored-by: Varixo <[email protected]> Co-authored-by: Miško Hevery <[email protected]>

wmertens force-pushed the custom-serialize branch from d7f7793 to 4d056d4 Compare March 17, 2025 18:15

Varixo approved these changes Mar 17, 2025

View reviewed changes

packages/qwik/src/core/signal/signal-api.ts Outdated Show resolved Hide resolved

feat(eslint): track missing signals in serializer update()

2944830

chore: rename the Signal class to SignalImpl

54acbc0

wmertens merged commit 4d5a0c5 into build/v2 Mar 18, 2025
19 checks passed

wmertens deleted the custom-serialize branch March 18, 2025 07:45

github-actions bot mentioned this pull request Mar 18, 2025

V2 Version Packages (alpha) #7392

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: custom serialization #7223

feat: custom serialization #7223

wmertens commented Jan 2, 2025 •

edited

Loading

changeset-bot bot commented Jan 2, 2025 •

edited

Loading

Varixo commented Jan 2, 2025

wmertens commented Jan 2, 2025

thejackshelton commented Jan 2, 2025

wmertens commented Jan 3, 2025

wmertens commented Jan 3, 2025

pkg-pr-new bot commented Jan 3, 2025 •

edited

Loading

github-actions bot commented Jan 3, 2025 •

edited

Loading

Varixo Feb 7, 2025

thejackshelton left a comment

thejackshelton Feb 7, 2025

wmertens Feb 13, 2025

thejackshelton Feb 7, 2025

wmertens Feb 13, 2025

thejackshelton-kunaico left a comment •

edited

Loading

ianlet commented Feb 13, 2025 •

edited

Loading

ianlet commented Feb 13, 2025 •

edited

Loading

wmertens commented Feb 13, 2025

wmertens commented Feb 13, 2025

ianlet commented Feb 14, 2025 •

edited

Loading

ianlet commented Feb 14, 2025 •

edited

Loading

mhevery left a comment

Varixo left a comment

wmertens commented Mar 18, 2025

feat: custom serialization #7223

feat: custom serialization #7223

Conversation

wmertens commented Jan 2, 2025 • edited Loading

TODO

changeset-bot bot commented Jan 2, 2025 • edited Loading

🦋 Changeset detected

Varixo commented Jan 2, 2025

wmertens commented Jan 2, 2025

thejackshelton commented Jan 2, 2025

wmertens commented Jan 3, 2025

wmertens commented Jan 3, 2025

pkg-pr-new bot commented Jan 3, 2025 • edited Loading

github-actions bot commented Jan 3, 2025 • edited Loading

built with Refined Cloudflare Pages Action

⚡ Cloudflare Pages Deployment

Varixo Feb 7, 2025

Choose a reason for hiding this comment

thejackshelton left a comment

Choose a reason for hiding this comment

thejackshelton Feb 7, 2025

Choose a reason for hiding this comment

wmertens Feb 13, 2025

Choose a reason for hiding this comment

thejackshelton Feb 7, 2025

Choose a reason for hiding this comment

wmertens Feb 13, 2025

Choose a reason for hiding this comment

thejackshelton-kunaico left a comment • edited Loading

Choose a reason for hiding this comment

ianlet commented Feb 13, 2025 • edited Loading

ianlet commented Feb 13, 2025 • edited Loading

wmertens commented Feb 13, 2025

wmertens commented Feb 13, 2025

ianlet commented Feb 14, 2025 • edited Loading

ianlet commented Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

PR Overview

Reviewed Changes

mhevery left a comment

Choose a reason for hiding this comment

Varixo left a comment

Choose a reason for hiding this comment

wmertens commented Mar 18, 2025

wmertens commented Jan 2, 2025 •

edited

Loading

changeset-bot bot commented Jan 2, 2025 •

edited

Loading

pkg-pr-new bot commented Jan 3, 2025 •

edited

Loading

github-actions bot commented Jan 3, 2025 •

edited

Loading

thejackshelton-kunaico left a comment •

edited

Loading

ianlet commented Feb 13, 2025 •

edited

Loading

ianlet commented Feb 13, 2025 •

edited

Loading

ianlet commented Feb 14, 2025 •

edited

Loading

ianlet commented Feb 14, 2025 •

edited

Loading