Add support for WebGPURenderer and TSL example

[vincentfretin]

Description:

For those that want to play with WebGPURenderer and TSL.
Run

npm run dist # to build dist/aframe-master.module.min.js used in tsl example
npm run start:webgpu

Start from
https://localhost:8080/showcase/tsl/
and port an example from https://threejs.org/examples/?q=tsl

Changes proposed:

• Rename a-node isNode to isANode, this was the cause of infinite loop with build function on WebGPURenderer when building a shader
• Add a start:webgpu command that will import from three/webgpu instead of three via a webpack alias
• Set dummy renderer.xr.setPoseTarget for now when it's WebGPURenderer
• Import UniformsUtils and UniformsLib from three/src (used by sdf/msdf text shaders), those are not exposed on the three/webgpu build
• Fix vendor/rStats.extras.js where renderer.info.programs is undefined with WebGPURenderer

All the changes here are backward compatible with the default WebGLRenderer umd aframe build.

You can use WebGPURenderer in two ways:

• via an importmap defining three and three/webpgu to use three.webgpu.js instead of three.module.js

• in your webpack project defining the correct alias:

  resolve: {
    alias: {
      'three$': path.resolve(__dirname, 'node_modules/three/build/three.webgpu.js')
    }
  };

and using in package.json this PR for now:

"dependencies": {
  "aframe": "vincentfretin/aframe#webgpu"
}

Using WebGPURenderer with classic aframe umd build is not supported, we don't generate a build for that.

[dmarcos]

We should def understand what's going on here

[vincentfretin]

The infinite loop was here https://github.com/mrdoob/three.js/blob/a5734130516db50c1de2cdfe27f477a33ecd1c87/src/nodes/core/Node.js#L640-L648
I don't know much about the new node system to say what that code is doing.

[vincentfretin]

As I said in #5363 (comment) my guess is that in aframe we set a reference to the entity with el on a three object.

Doing a search I get the following places:

$ ag "\.el = " src/
src/core/system.js
31:  this.el = sceneEl;

src/core/component.js
66:  this.el = el;

src/core/a-entity.js
36:    this.object3D.el = this;
111:    this.object3D.el = null;
140:    obj.el = this;
143:        child.el = self;

src/components/material.js
112:    shaderInstance.el = this.el;

src/components/light.js
205:      this.light.el = el;

src/components/text.js
437:  shaderObject.el = el;

My guess is that the one set on shaderInstance.el is the culprit and end up in the shader/material being build, but not sure how, I'm not familiar with the new webgpu code path.

[vincentfretin]

I'm not aware of any community components that is using an isNode check, I think it's fine to rename it to isANode and add a note about it in the release notes.

[dmarcos]

Cool. I would postpone this to 1.8.0. I will be cool to have a WebGPU path.

[vincentfretin]

sdf/msdf text is completely broken
WebGLRenderer

WebGPURenderer

[vincentfretin]

http://127.0.0.1:8080/docs/basic-scene/
2 errors in console: NodeMaterial: Material "ShaderMaterial" is not compatible.
may be related to the text shader

http://127.0.0.1:8080/showcase/curved-mockups/
background image is black on Chrome Android where is really run on webgpu. On desktop it's okay but it fallbacks to webgl2 on Ubuntu, so not sure if it's a webgpu on mobile issue.

[vincentfretin]

http://127.0.0.1:8080/boilerplate/hello-world/
WebGLRenderer

WebGPURenderer (webgpu backend or webgl2 fallback with r173)

[arpu]

for me the only question is, what is the usecase for renderer.xr.setPoseTarget i never need this and its only used at

aframe/src/core/scene/a-scene.js

Line 553 in b9e4838

if (this.camera) { renderer.xr.setPoseTarget(this.camera.el.object3D); }

[vincentfretin]

@dmarcos I think we could add a TSL example and then merge this, all the changes here are backward compatible with the WebGLRenderer aframe umd build. We can then make other changes if needed, mainly the text component that is completely broken by replacing it with troika-text if that one work with WebGPU backend.

As of three r173, VR mode is not working with WebGPU, we get the error:
THREE.XRManager: XR is currently not supported with a WebGPU backend. Use WebGL by passing "{ forceWebGL: true }" to the constructor of the renderer.

[vincentfretin]

for me the only question is, what is the usecase for renderer.xr.setPoseTarget i never need this and its only used at

aframe/src/core/scene/a-scene.js

Line 553 in b9e4838

if (this.camera) { renderer.xr.setPoseTarget(this.camera.el.object3D); }

Good question, we have that patch supermedium/three.js@3bfa95d (in 0.173.4 branch)
not sure what it does, maybe keep the xz position of the aframe camera when entering VR? The y position is overridden in VR that I know. I usually set a xz position on camera rig, not the camera.

[mrxz]

for me the only question is, what is the usecase for renderer.xr.setPoseTarget i never need this and its only used at

@arpu In Three.js the WebXRManager moves the "user" camera to match the xr camera. In A-Frame the camera is conceptually an <a-camera> or <a-entity camera>, which has the actual camera as a child. Just like how components such as look-controls and wasd-controls move the camera entity and not the camera itself, the setPoseTarget is there to let Three.js do the same.

[vincentfretin]

So I guess we will need to do a similar patch to XRManager (used by WebGPU backend) to implement setPoseTarget when VR will be supported.

[mrxz]

There is an alternative which I use in my fork, namely disabling XR camera updates in Three.js and doing it ourselves in the camera component. Not only does this avoid needing a modification to Three.js, it also ensures that the camera is up-to-date after the camera component. So any logic that is based on the users head position/orientation can use after: ['camera']. Currently these components tend to use 1 frame stale data.

[dmarcos]

for me the only question is, what is the usecase for renderer.xr.setPoseTarget i never need this and its only used at

@arpu In Three.js the WebXRManager moves the "user" camera to match the xr camera. In A-Frame the camera is conceptually an <a-camera> or <a-entity camera>, which has the actual camera as a child. Just like how components such as look-controls and wasd-controls move the camera entity and not the camera itself, the setPoseTarget is there to let Three.js do the same.

Correct. That method or similar existed in WebVRManager and WebXRManager and was removed at some point. Since the pose applies only to the camera and one cannot apply it to an arbitrary Object3D. I raised the concern to THREE but was ignored.

[dmarcos]

There is an alternative which I use in my fork, namely disabling XR camera updates in Three.js and doing it ourselves in the camera component. Not only does this avoid needing a modification to Three.js, it also ensures that the camera is up-to-date after the camera component. So any logic that is based on the users head position/orientation can use after: ['camera']. Currently these components tend to use 1 frame stale data.

How do you disable camera updates?

[mrxz]

How do you disable camera updates?

By setting renderer.xr.cameraAutoUpdate to false (see https://github.com/mrdoob/three.js/blob/dev/src/renderers/WebGLRenderer.js#L1160). But as with anything, it isn't without its drawbacks. If the camera component handles the updating, then any camera movement afterwards (directly or indirectly through a camera rig) needs to be propagated to the world matrices of the sub cameras. This only needs to be done right before rendering, but it no longer happens automatically.

Basically the above flag is an all-or-nothing and in an ideal world the part that transfers the WebXR pose onto the camera is done in the camera component and the part that updates the world matrices of the underlying eye cameras is done right before rendering.

But I have no idea if the XRManager for WebGPURenderer is going to have the same behaviour, probably best to wait and see how that implementation will behave.

[dmarcos]

How do you disable camera updates?

By setting renderer.xr.cameraAutoUpdate to false (see https://github.com/mrdoob/three.js/blob/dev/src/renderers/WebGLRenderer.js#L1160). But as with anything, it isn't without its drawbacks. If the camera component handles the updating, then any camera movement afterwards (directly or indirectly through a camera rig) needs to be propagated to the world matrices of the sub cameras. This only needs to be done right before rendering, but it no longer happens automatically.

Basically the above flag is an all-or-nothing and in an ideal world the part that transfers the WebXR pose onto the camera is done in the camera component and the part that updates the world matrices of the underlying eye cameras is done right before rendering.

But I have no idea if the XRManager for WebGPURenderer is going to have the same behaviour, probably best to wait and see how that implementation will behave.

Thanks. I like the idea of having more control of the lower level parts that concern XR. So in your case the pose used for rendering is not the most current one since it’s done in the component?