For reference: glTF Extension `KHR_gaussian_splatting` for 3DGS scene representation

[jo-chemla]

Description

Following thread NASA-AMMOS/3DTilesRendererJS#863 I just opened, @gkjohnson suggested I should cross-post it directly on the three.js repo. Feel free to close this and anyone interested can watch the universal splat thread instead - link below - since this 3DGS feature might be out-of-scope for the threejs library at the moment.

Solution

Here mkkellogg/GaussianSplats3D#47 is a mega discussion regarding a universal gaussian-splat format, with a lot of great takes and mentions - mostly regarding compression, like Niantic SPZ, and standardization.

An interesting recent comment by lilleyse (Cesium team member) redirects to the draft KHR_gaussian_splatting glTF extension as well as an implementation within CesiumJS in the experimental splat-shader branch. This extension adds _ROTATION and _SCALE attributes to a glTF POINTS, so these can be used to represent a gaussian-splat scene. If or once that standard settles and makes its way to official KHR_extension, it would be uesful for threejs to interpret this extension and give access to these extra attributes.

The Cesiumjs implementation extends that KHR_gaussian_splatting extension to a HLOD (Hierarchical LoD) tileset container of mesh scenes, OGC-3D-tiles, where every leaf node tile is a glTF scene that supports that gaussian-splat extension.

Alternatives

CesiumJS viewer will probably be the first reference implementation for that KHR_gaussian_splatting extension + OGC-3D-tiles

Additional context

Following the original release of the 3DGS paper by INRIA team at SIGGRAPH 2023, there have been multiple JS renderers and viewers implemented, some based on threejs like mkkellogg/GaussianSplats3D and some others which aren't, eg antimatter15/splat lumalabs or babylonjs.

[gkjohnson]

cc @donmccurdy do you happen to have any sense for where this is headed? And as mentioned in the other issue - how we render multiple separate splat objects when splats all have to be sorted together before rendering remains an open question. I'm not exactly how that would work here.

[DennisSmolek]

I attended the Khronos talk that (briefly) went over this at Siggraph/Asia 2024 and it's still super up in the air. I know Cesium has some working plans but there was talk on even the correct way to store or reference the splats.

I'm totally down for them to both be in both GLTF and three but it might be a bit before that gets solidified.

[donmccurdy]

Perhaps the place to start would be to figure out what a useful gaussian splat representation in three.js (three/addons/*? separate package?) could/should look like, and whether it's renderer-agnostic or reliant on WebGPURenderer/TSL. Without that, and without much certainty on the glTF extension's roadmap, it might be too early to think about support in GLTFLoader. It is possible to implement extensions for GLTFLoader without modifying the loader, as well.

[cavencj]

has any plan to do this ?

[gkjohnson]

I posted this in another issue but this area still seems to be evolving a lot. Here's a new paper with a new rendering approach aimed at solving some of the issues with splats:

https://radfoam.github.io/

https://x.com/taiyasaki/status/1886625119851962598

I'm wondering if there are other techniques that are going to supersede gaussian splatting in the next year or two

[jo-chemla]

Quick note following great progress by 3DGS community - still evolving pretty fast indeed, towards standardization - the current route chosen by some orgs seem to be to release a glTF extension, KHR_spz_gaussian_splats_compression KhronosGroup/glTF#2490 which is currently being split into two extension, one base and one specific to niantic's spz compressed format to allow easier support for other compression formats.

They have successfully built a 3Dtiles tileset of 3DGS tiles either KHR_gaussian_splatting or KHR_spz_gaussian_splats_compression, and converter from massive 3Dgs to tileset of splats.

[Junior2Ran]

I think Gaussian Splatting is still a powerful technique for rendering realistic scenes in current days, andKHR_spz_gaussian_splats_compression seems to be a useful extension. Maybe it can be added to GLTFLoader when the extension is stable.