Making the `<model>` element compatible with modern web frameworks and development paradigms.

[trusktr]

Copilot Issue Summary

click to expand

Description:

The author, trusktr, proposes making the <model> HTML element more compatible with modern web frameworks like React, Vue, Svelte, and Solid. The current implementation of the <model> element, specifically its getCamera() API which returns a JavaScript object instead of an HTMLElement, is not compatible with DOM manipulation tools used by these frameworks. This requires developers to create unnecessary wrappers for each framework to bridge the gap, making the code less maintainable.

Proposed Solutions:

1. DOM-Based API:

   • Introduce a more DOM-centric approach where cameras and other 3D elements can be defined and manipulated directly in the DOM.
   • Example:

     <model>
       <camera type="perspective" position="10 20 30"></camera>
     </model>

2. Camera Attributes on <model> Element:

   • Alternatively, allow camera properties to be set as attributes on the <model> element.
   • Example:

     <model camera="perspective" camera-position="10 20 30"></model>

3. Further Future Development:

   • Decouple <camera> and <model> elements to be used within a <scene> element, allowing for more complex 3D scenes.
   • Example:

     <scene fog="linear" fog-color="pink">
       <camera type="perspective" position="..."></camera>
       <model rotation="..."></model>
       <model scale="..."></model>
       <box></box>
       <sphere></sphere>
     </scene>

Discussion Points:

• Ensuring compatibility with existing web frameworks.
• Avoiding the need for custom wrappers and simplifying developer workflow.
• Maintaining a DOM-centric approach for better integration with web tools.
• Potential benefits of a standardized 3D API for the web.

The author invites further discussion and collaboration to ideate a future where 3D elements are seamlessly integrated into the web ecosystem with high compatibility and simplicity.

Original content

Hello! I first wrote about this here.

https://twitter.com/trusktr/status/1717795902272974942

I want to open a conversation on how we could make the <model> element more compatible with today's web frameworks, and how currently it is not.

Note

I work on Lume, HTML elements for 3D rendering, and I have been spent a lot of time imagining what native 3D elements in browsers could be like.
Try viewing and hovering on elements in the devtools element inspector when view examples.

I'll re-post what I wrote on Twitter here, updated and improved a little:

---

@apple's HTML <model> element in #visionOS Safari is incredibly neat (see the video, at 12:36), hinting at an awesome future of high-level 3D web capabilities that don't require intimate programming knowledge to use.

It however does not hint at a full HTML 3D future compatible with React, Vue, Svelte, Solid, and all the other DOM tools we've evolved over the years. Let me explaine why, and perhaps we can fix this.

<model> enables amazing native features for web in a simple way, and starts to paint an awesome future for web, but I believe we can plan our HTML future better based on the present (cc @jensimmons as the curator of our #webdev needs for @WebKit, and @marcoscaceres as editor of the spec):

The current model.getCamera() API returns a non-HTMLElement plain-JavaScript object, which means all of our DOM-manipulation tools (@buildWithLit, @reactjs, @vuejs, @sveltejs, @solid_js, and more) will not be able to manipulate a <model> element's camera out of the box using their declarative-reactive templating systems, requiring a non-standard new way of programming interactivity specifically for <model>. Web devs will build unnecessary wrappers (for every framework!) to bridge this gap, in order to tie <model> camera interactivity back into our paradigms.

As an example of how unnecessary wrappers would be, react-three-fiber wraps Three.js to add the declarative-reactive support for Three.js in React because Three.js is not DOM-based (this is not unnecessary for Three.js, but <model> aims to be a DOM API and makes part of its API necessarily require non-DOM wrappers). The wrappers that people make for <model> will need to map attributes/properties from their templates to the <model>'s non-DOM camera objects.

With HTML elements (and well-designed Custom Elements), there is no need for wrappers to be made for every framework, all elements typically work out of the box in any web framework (there are a few edge cases, but we've learned to avoid those in design of new HTML elements).

The following HTML interface is an example that would be compatible with the DOM future all frameworks have been building:

<model ...>
  <camera type="perspective" position="10 20 30"></camera>
</model>

This is compatible because by making the camera configurable via DOM with HTML support, every web framework can manipulate it out of the box (and not just on the client side, but also on the server side!).

Alternatively, camera attributes on the <model> element would also be a compatible pattern, and could later be superceded with HTML elements that override the attribute behavior:

<model camera="perspective" camera-position="10 20 30" etc></model>

These designs are compatible with all of today's frameworks because they embrace the DOM tree model that all DOM tools are designed to work with.

The getCamera() API, though extremely nifty for what it can do, is not compatible because it side-steps the DOM trees that these tools manipulate, which will make our code more WET (less DRY) due to those wrappers we'll need for every framework to connect to <model> element features (cameras). If the API were to give us elements and attributes (which elements, and which attributes, can be debated), we'd get these connections concisely and for free.

Imagine <model> comes out in all browsers, and out-of-the-box anyone using any framework can manipulate all aspects of a <model> scene right away without escaping out of their framework to write custom JavaScript.
 
With a DOM-based API, the Promise return would no longer be necessary for getCamera(), as these plain JS objects will be controlled while their rendering effects are abstracted underneath. The camera DOM can internally proxy to an underlying renderer (out of process or not).

Today, most frameworks practice one-way data flow (except in edge cases as with <input>s, etc), but the getCamera() API, besides not being compatible with DOM tools, is a more cumbersome two-way data flow design:

1. First, the user has to get a Promise and wait for it. With DOM API, users never have to wait for sub-objects (children) to be ready by using Promises returned by the browser. For scenarios involving elements that load assets (f.e. <img>), browsers fire a load event when the element has loaded an asset or assets.
2. Then the user can start to interact with a camera object but not in a standard way (JS only, no compat with tools).

Instead, with a DOM-centric approach, users would be able to simply map their desired state to the DOM (whether attribtues on <model> or a new element like <camera>). The user does not need to know when the underlying internal camera is ready for rendering, just like the user does not need to know when a <select> is ready for rendering: they simply map their state to the DOM, and they expect the display to update. load events can be emitted to tell users when assets are ready. Promise is not necessary.

The current getCamera() essentially creates a new and alternative object model that requires new tooling for all frameworks, yet does not provide a clear benefit that would make this tradeoff worth it.

We can also get creative here, while not veering away from the really great web developer paradigms we already have:

In a further future (and based on my experience making Lume, and knowledge of A-Frame, Trois for Vue, Threlte for Svelte, react-three-fiber for React, and other similar libraries for various frameworks), <camera> and <model> can be decoupled from each other, used in a 3D <scene>, implicitly disabling the <model>'s camera and instead rendering relative to a user-defined <scene> with custom graphics interspersed with any number of <model> elements:

<scene fog="linear" fog-color="pink" ...>
  <camera type="perspective" position="..."></camera>
  <model rotation="..." ...></model>
  <model scale="..." ...></model>
  <box ...></box>
  <sphere ...></sphere>
</scene>

Such a scene as above is fully compatible with web tools that have evolved over so many years of thought.

Changing the <model> API would be easy, but breaking apps would be extremely difficult. This is why we need to imagine, right now: how do we ensure that the future brings us elements that immediately fit right in with the tools we've worked hard to create?

I would love to help ideate this future. I've started to ideate over at https://lume.io.

I would love to continue this conversation to ensure that we can create a set of 3D elements that meet the needs of the vast majority of web developers in the simplest way possible with highest compatibility with the existing ecosystem.

[marcoscaceres]

It's definitely worth exploring the design of the API (e.g., #12 already raised the issue around the promises and #41 and a ton of other issues).

I think the problem here is that this is conflating what the format provides and what model provides. The <model> element, as it stands, it just a simple wrapper for some 3D format (USDZ or gLTF) so that a 3D file can be rendered on a page.

In designing an API, what we need to figure out is exactly what overlap there is between the two formats in accessing the common internals (e.g., they may both have an initial camera position defined in the format itself). We can't really do much more than that, I think... but that's still allows for a lot of stuff, as model-viewer has shown.

I don't think we want to end up with <box ...></box> <sphere ...></sphere> etc. We'd basically be recreating VRML. USD and gLTF already specify all the 3D object/world information, as well as textures, etc. To do that again in HTML would be recreating the wheel, which few definitely don't want to do.

I guess what would be nice with existing things on the Web is if there was some way to simply light a scene using environmental lighting information and present it in a stereoscopic view (like what VisionOS does), but take away the ability for the page to capture the canvas (for privacy, as the UA would be reflecting the user's environment).

I don't know if that's possible/feasible, but it seems that's what you'd want here: to continue to support all the legacy content, but just have it be rendered/lit correctly in the new context.

At the same time, it's nice to just put a <model> on the page, just like one does with an <img> tag and have it do simple things... for anything more fancy, there is WebGPU, WebGL, WebXR, Canvas etc.

[zachernuk]

I totally agree about establishing some manner of composing and manipulating model contents - I'm just conscious that the more that goes into an initial specification, the harder it becomes to agree upon it and the more implementation work must be done before it's complete. To that end, in the interests of finding the absolute MVP, I'm suggesting that we defer any scene inspection until a post-V1. @trusktr Would you be okay to close this?

[trusktr]

I'm suggesting that we defer any scene inspection until a post-V1

I agree, however it has already been decided, it seems, that the camera should be controllable by the user in the MVP

So, if camera control will ship in the MVP, it would be great to do it the right way so that it will fit smoothly into frameworks (react, vue, svelte, solid, angular, custom elements, etc), without instead being an awkward API everyone has to escape out of the framework to use.

Here are some concrete problems:

• server side rendering: if the camera is controlled with JavaScript only, that means an HTML-outputting server cannot send HTML to the client to tell the client where the camera should be!
• This also means if JavaScript is disabled, camera control will not even work!
• The JS-only camera API means that accessibility readers read HTML will not see camera information
• When the Web Components CG releases HTML Modules and Declarative Custom Elements (write declarative-reactive elements purely in HTML), the <model> element will not have a controllable camera without having to escape out of the declarative paradigm into JavaScript.
  • Declarative Custom Elements also will allow programs in a multitude of programming languages (PHP, JavaScript, Python, Rust, C/C++, Go, etc) to serve Custom Element definitions as HTML to clients, but camera control of a <model> element will not fit into design.

By skipping out on HTML for some of <model>'s features, the above scenarios will require implementing custom and bespoke escape hatches to access JavaScript to control a camera or to read camera state. For example,

• frameworks will need to tell the user to use refs to write custom JS, and these refs will not be SSR compatible
• server-side systems will need to output JavaScript instead of only HTML which will get messy
• accessibility readers will need to add JavaScript support to access a model's camera
• search engines will not be able to index camera information without using JavaScript
  • keeping in mind that in the future this could be possible: <camera ...><my-hud></my-hud></camera> which could include indexable content including text, more on that in the next comment after this one.
• etc

etc

The current JavaScript API is just on a path that is not aligned with our existing built-in, and userland web development, paradigms (namely, declarative-reactive paradigms). Even old frameworks like jQuery are designed to manipulate elements and attributes. Elements and attributes are the ideal fabric to support out of the box without first adding APIs that circumvent this.

So we have to ask ourselves: are we really ok with this design for camera control that has no alignment at all with all the tooling we've created over decades?

Possible paths for an MVP

1) replace (or provide alternative to) the JavaScript-only API with HTML

Replacing the API with an HTML interface would be ideal, as it would eliminate the dancing with promises. The promises are merely a leaked implementation detail. However, if it is too difficult to remove the JS API, then the HTML can be provided as the ideal recommended way to interact with the camera once the element lands in all browsers.

Promises are out of pattern with other elements that emit load events for that purpose, and we should follow those patterns for consistency. F.e. <img> elements: if we want to access their rendering features (f.e. pixels) then we wait for load. If we want to know when rendering features of a <model> are ready, we could similar just wait for a load event. (People can make promise wrappers for load events on their own if they want, that's easy, but I discourage that because load events can repeat and promises cannot, and repeatable promise patterns are quite messy).

The camera (and any features that are currently controlled with JavaScript) should ideally be controllable via HTML as attributes or as elements (it doesn't matter if the rendering of the camera happens later in the implementation, or even out of process, and a load event would be standard for that if and only if needed).

The following examples are basically reprensentative of what is needed to go in a direction that aligns with all the frameworks we've already made, plus the future of web which will likely include Declarative Custom Elements (custom elements written purely in HTML, planned as part of HTML Modules):

<model ...>
  <camera type="perspective" position="10 20 30"></camera>
</model>

or

<model camera="perspective" camera-position="10 20 30" etc></model>

This is a simple modification to the <model> API. Instead of (or as an additional alternative to) the JavaScript API, providing something that everyone in every framework (client or server) can use.

2) remove camera control for MVP, come back to it after

If the MVP will remove the camera controls, so that an HTML interface can be properly spec'd later, giving more time for conversation, that would be better (to me) than an API that we know is already not aligned with how elements are designed in general, especially with the large amount of the Web Components community which we should tap into.

---

If you want to get <model> into spec and out in all browsers more quickly, with back compat for any apps that already use camera controls, then I think that option 1 is the way to go.

But please, please, please, don't willingly release something that we know is not aligned with standard paradigms, its just going to cement a way of doing things that we know is not how web developers do things.

We, as web developers, don't want our browser to battle against the tools we know, and even love. We want browsers to support us in the way we've already established (which browsers helped get us to). In a sense, <model> is going against the grain of even the very own elements of browsers themselves by providing this new promise-based API instead of standard HTML and load events.

[trusktr]

I don't think we want to end up with <box ...></box> <sphere ...></sphere> etc. We'd basically be recreating VRML.

For the use case of <model>, perhaps not. For the use case od 3D in the web, totally.

So, I think it would be ideal if we keep this in mind, even if it is not part of the <model> element. The reason is because, then some features can be standardized later. For example, suppose that this is the model MVP:

<model ...>
  <camera type="perspective" position="10 20 30"></camera>
</model>

Later on, when browsers have had time to work on their 3d rendering and ironed out all the issues, new ways of making scenes outside of the <model> element could be introduced, and a feature like <camera> could be easily re-used:

<gpuscene>
  <box position="1 2 3"></box>

  <!-- the same camera element people already know how to use: -->
  <camera type="perspective" position="10 20 30"></camera>
</gpuscene>

Later on, we could even decide what it means to put a <model> inside a 3D scene! For example:

<gpuscene>
  <box position="1 2 3"></box>

  <!-- same model element people already know!: -->
  <model ...></model>

  <!-- the same camera element people already know how to use: -->
  <camera type="perspective" position="10 20 30"></camera>
</gpuscene>

Keeping this in mind will help us get to that eventual future which some of us do want, even if that's outside of the <model> element's scope.

I know I really do, and I know many people can benefit from it.

In Lume I'm working on answering the question "what might built-in 3D HTML elements look like if future browsers had them?":

It would be amazing if Lume/A-Frame/X3D/threlte/react-three-fiber/etc all became obsolete projects because browsers one day gave us the power of built-in 3D HTML elements.

[trusktr]

I added Copilot summary to the top of the OP, which I'll keep updated over time.

[zachernuk]

even if it is not part of the <model> element.

I think that's the core question for what is and isn't part of <model> - we all agree that there should be a way to create a spatial web, with composable elements, potentially positioned by constraints etc. I also think that it'll be possible to explore aspects of that spatial web with <model> - but 1. doesn't need to be Model's primary function and, by consequence 2. should place API obligations on Model to do more than it's supposed to.

This is not solely up to me, but the way I've been thinking of it as representing a single file of 3D content at a time, portrayed and manipulated monolithically. In 2024 I revised the control mechanism to remove the idea of Camera, with a preference to depict the transform on the Model entity as (very tentatively) the entityTransform, described in the explainer.

Because Model is intended to act as the simplest closure over 3D content - similar to <img> in 2D - it's not expected to provide analogs to more expressive contexts like <svg> or <canvas>, in either its CanvasRenderingContext2D flavor or the mode modern WebGLRenderingContext and WebGPU flavors. Those remain highly desirable collections of capability to pursue - I just don't think they have to come through (or start before) statically-provided, monolithic 3D asset content.