For reference: glTF Extension KHR_gaussian_splatting for 3DGS scene representation

[jo-chemla]

Description

Following thread NASA-AMMOS/3DTilesRendererJS#863 I just opened, @gkjohnson suggested I should cross-post it directly on the three.js repo. Feel free to close this and anyone interested can watch the universal splat thread instead - link below - since this 3DGS feature might be out-of-scope for the threejs library at the moment.

Solution

Here mkkellogg/GaussianSplats3D#47 is a mega discussion regarding a universal gaussian-splat format, with a lot of great takes and mentions - mostly regarding compression, like Niantic SPZ, and standardization.

An interesting recent comment by lilleyse (Cesium team member) redirects to the draft KHR_gaussian_splatting glTF extension as well as an implementation within CesiumJS in the experimental splat-shader branch. This extension adds _ROTATION and _SCALE attributes to a glTF POINTS, so these can be used to represent a gaussian-splat scene. If or once that standard settles and makes its way to official KHR_extension, it would be uesful for threejs to interpret this extension and give access to these extra attributes.

The Cesiumjs implementation extends that KHR_gaussian_splatting extension to a HLOD (Hierarchical LoD) tileset container of mesh scenes, OGC-3D-tiles, where every leaf node tile is a glTF scene that supports that gaussian-splat extension.

Alternatives

CesiumJS viewer will probably be the first reference implementation for that KHR_gaussian_splatting extension + OGC-3D-tiles

Additional context

Following the original release of the 3DGS paper by INRIA team at SIGGRAPH 2023, there have been multiple JS renderers and viewers implemented, some based on threejs like mkkellogg/GaussianSplats3D and some others which aren't, eg antimatter15/splat lumalabs or babylonjs.

[gkjohnson]

cc @donmccurdy do you happen to have any sense for where this is headed? And as mentioned in the other issue - how we render multiple separate splat objects when splats all have to be sorted together before rendering remains an open question. I'm not exactly how that would work here.

[DennisSmolek]

I attended the Khronos talk that (briefly) went over this at Siggraph/Asia 2024 and it's still super up in the air. I know Cesium has some working plans but there was talk on even the correct way to store or reference the splats.

I'm totally down for them to both be in both GLTF and three but it might be a bit before that gets solidified.

[donmccurdy]

Perhaps the place to start would be to figure out what a useful gaussian splat representation in three.js (three/addons/*? separate package?) could/should look like, and whether it's renderer-agnostic or reliant on WebGPURenderer/TSL. Without that, and without much certainty on the glTF extension's roadmap, it might be too early to think about support in GLTFLoader. It is possible to implement extensions for GLTFLoader without modifying the loader, as well.