Add morph targets

[nicopap]

Objective

• Add morph targets to bevy_pbr (closes GLTF Morph Targets #5756) & load them from glTF
• Supersedes Load morph target buffers from GLTF assets #3722
• Fixes Cannot play animation correctly. #6814

Morph targets (also known as shape interpolation, shape keys, or blend shapes) allow animating individual vertices with fine grained controls. This is typically used for facial expressions. By specifying multiple poses as vertex offset, and providing a set of weight of each pose, it is possible to define surprisingly realistic transitions between poses. Blending between multiple poses also allow composition. Morph targets are part of the gltf standard and are a feature of Unity and Unreal, and babylone.js, it is only natural to implement them in bevy.

Solution

This implementation of morph targets uses a 3d texture where each pixel is a component of an animated attribute. Each layer is a different target. We use a 2d texture for each target, because the number of attribute×components×animated vertices is expected to always exceed the maximum pixel row size limit of webGL2. It copies fairly closely the way skinning is implemented on the CPU side, while on the GPU side, the shader morph target implementation is a relatively trivial detail.

We add an optional morph_texture to the Mesh struct. The morph_texture is built through a method that accepts an iterator over attribute buffers.

The MorphWeights component, user-accessible, controls the blend of poses used by mesh instances (so that multiple copy of the same mesh may have different weights), all the weights are uploaded to a uniform buffer of 256 f32. We limit to 16 poses per mesh, and a total of 256 poses.

More literature:

• Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/
• Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE
• GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits
• Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772

bevy_morph_targets-2023-04-11.mp4

bevy_morph_targets-2023-05-10.mp4

Acknowledgements

• Thanks to storytold for sponsoring the feature
• Thanks to superdump and james7132 for guidance and help figuring out stuff

Future work

• Handling of less and more attributes (eg: animated uv, animated arbitrary attributes)
• Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses)
• Better animation API, see AnimationPlayer can play unrelated AnimationClips  #8357

---

Changelog

• Add morph targets to bevy meshes
  • Support up to 64 poses per mesh of individually up to 116508 vertices, animation currently strictly limited to the position, normal and tangent attributes.
  • Load a morph target using Mesh::set_morph_targets
• Add VisitMorphTargets and VisitMorphAttributes traits to bevy_render, this allows defining morph targets (a fairly complex and nested data structure) through iterators (ie: single copy instead of passing around buffers), see documentation of those traits for details
• Add MorphWeights component exported by bevy_render
  • MorphWeights control mesh's morph target weights, blending between various poses defined as morph targets.
  • MorphWeights are directly inherited by direct children (single level of hierarchy) of an entity. This allows controlling several mesh primitives through a unique entity as per GLTF spec.
• Add MorphTargetNames component, naming each indices of loaded morph targets.
• Load morph targets weights and buffers in bevy_gltf
• handle morph targets animations in bevy_animation (previously, it was a warn! log)
• Add the MorphStressTest.gltf asset for morph targets testing, taken from the glTF samples repo, CC0.
• Add morph target manipulation to scene_viewer
• Separate the animation code in scene_viewer from the rest of the code, reducing #[cfg(feature)] noise
• Add the morph_targets.rs example to show off how to manipulate morph targets, loading MorpStressTest.gltf

Migration Guide

• (very specialized, unlikely to be touched by 3rd parties)
  • MeshPipeline now has a single mesh_layouts field rather than separate mesh_layout and skinned_mesh_layout fields. You should handle all possible mesh bind group layouts in your implementation
  • You should also handle properly the new MORPH_TARGETS shader def and mesh pipeline key. A new function is exposed to make this easier: setup_moprh_and_skinning_defs
  • The MeshBindGroup is now MeshBindGroups, cached bind groups are now accessed through the get method.

[nicopap]

Todo when merged: open issue detailing how paused AnimationPlayers overwrite user-set weights.

[nicopap]

Ready for review. @james7132 and @willstott101 might be interested in this.

[nicopap]

Note that this is duplicated in prepass.wgsl because I didn't want to move the struct Vertex definition to mesh_types.wgsl (which probably should be the place where Vertex is defined)

[nicopap]

I suspect splitting this in two different components is preferable. Otherwise it might become a bit awkward for the user to Query MorphWeights for specific mesh (as glTF mesh) or mesh (as bevy mesh)

[mockersf]

could you add an example dedicated to morphing, out of the scene viewer?

[nicopap]

@mockersf I can add a standalone example. Additionally, do you have any tips for avoiding code duplication between the example and the scene viewer?

[mockersf]

Additionally, do you have any tips for avoiding code duplication between the example and the scene viewer?

No, example scene viewer is duplicating code from other examples... If something is useful enough you can put it in bevy_animation

[cart]

Just pushed another "proposal commit": I moved morph target names to Mesh, which resolves my comment here: #8158 (comment)

[cart]

My latest change broke MorphViewerPlugin, but I'd like to discuss #8158 (comment) before fixing it because the code will be informed by that choice.

[nicopap]

I think morph target names are orthogonal to Mesh. It is not ideal to store them in the Mesh asset. Using a Arc<str> in a MorphTargetNames component seems to me a good way to avoid too many allocations.

[cart]

I think morph target names are orthogonal to Mesh. It is not ideal to store them in the Mesh asset. Using a Arc in a MorphTargetNames component seems to me a good way to avoid too many allocations.

I don't follow this logic. Can you explain how it is orthogonal? The Mesh is the canonical owner of the actual morph target data (by effectively owning the morph target image with the morph target data packed into it). Storing additional data that describes the morph targets (their names) makes perfect sense to me. The Mesh is "the thing" that is being morphed by morph targets. The GLTF data model nests the targetNames field right inside of the GltfMesh type. Why shouldn't our Mesh type own the names of the targets?

We do the same thing for "mesh vertex attribute data" by correlating their "mesh vertex attribute names". This feels conceptually almost identical.

[nicopap]

Hmm. Your reasoning is sensible. After all, the morph target image is stored on the Mesh.

Looks good.

[nicopap]

You can use the old test model here to try your changes. It has several meshes with morph targets, including Morph-Primitive Test:

• https://raw.githubusercontent.com/bevyengine/bevy/603956d39f7239ad3326f006bd86644014973e95/assets/models/animated/multiple_morph_target_meshes.gltf

It has more edge cases than MorphStressTest.gltf. See reasoning as to why it was replaced: #8158 (comment)

[cart]

I just removed the "nested propagated MorphWeights" in favor of duplicating curves within a clip. I think it keeps the mental model simpler by being truer to the current Bevy Mesh api. I also simplified the morph_targets example by making it play a specific clip (which is what actually games are going to do generally).

[nicopap]

Sure. I can finish work on your design to speed things up.

[nicopap]

I updated the scene_viewer/morph_viewer_plugin.rs to reflect the change to non-propagated morph weights.

Specifically pay attention to the add_targets function. Notice how as a user, I have to manually store the list of entities for a given instance of morph targets. Then use that list to iterate over all meshes related to a single instance of morph targets.

Manipulating individual morph targets programmatically, ie: not through a loaded glTF animation. Is a common usecase (dark souls character editor is an example). Similarly to how one would programmatically move bone transforms in a skinned mesh to handle IKs and ragdolls.

Try the scene_viewer with the Morph-Primitive-Test.glb I linked earlier. You'll notice the difference between managing morph weights across primitives of a single mesh and not managing them.

It's possible to write correct code. But if the user doesn't know about the subtle difference between a bevy mesh and a glTF mesh, they are going to be surprised when they update a MorphWeights component and see that only the fleshy part of the face moves, while the other part of the face with a different material stay fixed.

Note that it's fairly poor as a out-of-the-box experience. But it's possible to write a 3rd party plugin that adds a ManagedMorphWeights component that does what I've been describing.

[cart]

Hmmm ok I do see your point. I've also played around with morph targets a bit in Godot (which has the sync behavior). There are definitely arguments in favor of both approaches, but your "hierarchical sync" approach does have the benefits of "less user code for manual morph targets" and "more aligned with GLTF and Godot user expectations for morph target behavior". I guess either of those should outweigh "conceptual purity / Bevy alignment". This is especially true in light of the fact that Godot, Unity, and Unreal all have "mesh surface / primitives" concepts. Which means we will likely also want them, just to accommodate existing workflows. And then we'd have "bevy aligned / conceptual purity" and "behavior aligned with user expectations".

In short: you were right and sorry for the back and forth!

I'll revert things back to hierarchical sync (while keeping the other changes I made). I'll also do the "MorphWeights component split" to resolve some of the UX concerns with the hierarchy approach.

[JMS55]

Small non-blocking consideration: in order to write proper motion vectors, morph targets will need to be able to calculate their vertex position last frame during the current frame, which will involve knowing last frame's weights.

[cart]

I pushed the changes mentioned above. I also added MorphWeights::first_mesh() -> Option<&Handle<Mesh>> to make it easier to look up morph metadata stored in Mesh (such as the morph names). Still a bit of a compromise as we don't have a central "multi-surface Mesh" equivalent, but I still think this is preferable to storing names on individual entities.