KHR_gaussian_splatting and KHR_spz_gaussian_splats_compression

[weegeekps]

This extension proposal, KHR_spz_gaussian_splats_compression, allows for efficient storage of 3D Gaussian splats data within glTF using the SPZ compression library from Niantic Spatial. The extension is applied to a primitive. The SPZ binary blob is stored as a buffer within the glTF file, and implementations can use the SPZ library to either decompress and then map the compressed Gaussians into placeholder attributes on the primitive or directly decompress into their rendering pipeline if preferred. Content creators have the flexibility to choose to use no Spherical Harmonics, or up to all 3 degrees of spherical harmonics depending on their use case.

We are currently working on an implementation in the CesiumJS engine based on this draft that we hope to have released soon.

[DRx3D]

It appears that this extension is tied to the Niantic Spatial library. Is it necessary to specify a version or other unique identifier to ensure that the desired algorithm and calling sequence is used?

Also note that the license for the Niantic Spatial library is MIT.

[weegeekps]

Is it necessary to specify a version or other unique identifier to ensure that the desired algorithm and calling sequence is used?

Good question. The SPZ library packs along a version number within the binary data that we store in the buffer, so it's unnecessary to have a version number stored in the glTF metadata.

[weegeekps]

I took a first pass at splitting out the extension into a base extension and a SPZ extension. It's a rough draft but it should be a solid starting point for now. I did not just resurrect our original extension from late last year named KHR_gaussian_splatting and instead rewrote it based on the KHR_spz_gaussian_splats_compression draft.

Please let me know your thoughts and tweaks you'd like to see.

P.S. The min and max values in the accessors section is not fixed yet. It will be soon.

[GISUser1]

We developed a tool to convert a ply or splat file to KHR_spz_gaussian_splats_compression format 3dtiles. it is in our product GISBox and it is totally free to use. you guys can use it to test this new feature.

[weegeekps]

I just made a small change to the SPZ compression extension to (hopefully) simplify the language around inheritance from the base KHR_gaussian_splatting extension.

@lexaknyazev very interested in any thoughts you may have with this language.

[javagl]

I think that the KHR_gaussian_splatting extension should inculde a resolution for #2111. Athough that issue is still open an we have not explicitly agreed on "THE" solution that will be universally applied, I think that "namespacing" by prefixing the attribute names with KHR_gaussian_splatting sounds like a reasonable first shot. Even if there will be a different solution in the future, we could still have a solution for the current extension that is "safe", insofar that people who generate and read this data don't have to adjust their writer/reader code.

In the current version of the proposal, the accessors for these attribute do not define a byteOffset. This implies that the attributes have to be assumed to be stored in the buffer view in the order in which they are mentioned in the specs. I think that this should be clarified. Obsolete based on #2490 (comment)

[weegeekps]

I think that the KHR_gaussian_splatting extension should inculde a resolution for #2111.

I agree. I made a comment over in #2111 about a potential solution.

Regarding the byteOffset, do we need to explicitly specify that this extension supports them? Wouldn't this be covered by the base glTF specification since we don't explicitly specify that we don't support byte offsets?

[weegeekps]

I've been following what the OpenUSD folks are doing as well as talking with a few of them, and below are some proposed changes that I want to run by everyone before I make them to the specification. Some of these suggestions have been mentioned prior in other conversations, but I wanted to expand on them here and go into a bit more details now that I have a better understanding of them.

All of these additions are in the spirit of keeping this base extension forward looking.

Shape property

Researchers are currently looking a range of shapes for Gaussian splats. Today, we've seen research around the traditional ellipsoid shape, triangles, and quads. Adopting a property to specify what shape the Gaussians would be good for future-proofing.

Similar to what OpenUSD is doing, in the spec we would provide an initial list of some known options: ellipsoid, triangle, and quad. The default value will be ellipsoid. The property will be optional and omitting it will indicate the default value should be used.

We will likely need a mechanism for allowing custom values here. OpenUSD often is okay with stringly-typed enumerations, but within glTF we tend to be a bit more conservative.

Rendering Hints

The most significant differences between the Gaussian Splatting API that the OpenUSD folks are putting together and KHR_gaussian_splatting are best categorized as rendering hints. These are values that are known during training that must be passed along to the renderer to allow for proper rendering.

All of these hints would live within a hints object within the extension definition.

Like the shape property, it's impossible for us to know all of the possible future values for these. As hints, we'll define a "default" for each of these, and then explicitly state that implementers should fall back to the default if they come across a value they don't recognize, and attempt best effort at rendering.

It's also up to the renderer themselves if they use these hints for their renderers or not and use of them will be explicitly optional. This is in the same spirit as what OpenUSD is doing with these: the goal is to provide information that is useful to renderers but not make any assumptions as to whether or not those renderers use them.

Sorting Method

The sortingMethod specifies how the splats are sorted for the rendering process. The initial possible values match what OpenUSD is doing:

• cameraDistance which is the distance between the center of the splat and the position of the camera.
• zDepth which is the projected z-depth in the camera projection.

The default value will be cameraDistance.

Projection

The projection specifies how the splat shape is projected onto the image. Initial values diverge slightly from what OpenUSD is doing:

• perspective is your typical 3D perspective projection based on scene depth.
• orthographic projects the splat orthogonally into the image. OpenUSD refers to this as tangential today.

The default value will be perspective.

Sample with Shape and Rendering Hints

All of the fields proposed above are optional, but this is what it would look like with all of them defined.

"meshes": [{
    "primitives": [{
        "attributes": {
            "POSITION": 0,
            "COLOR_0": 1,
            "_SCALE": 2,
            "_ROTATION": 3,
            "_SH_DEGREE_1_COEF_0": 4,
            "_SH_DEGREE_1_COEF_1": 5,
            "_SH_DEGREE_1_COEF_2": 6
        },
        "mode": 0,
        "indices": 7,
        "extensions": {
            "KHR_gaussian_splatting": {
                "shape": "ellipsoid",
                "hints": {
                    "sortingMethod": "cameraDistance",
                    "projection": "perspective"
                }
            }
        }
    }]
}],

Including Spherical Harmonics in KHR_gaussian_splatting

With the Gaussian Splatting API that the OpenUSD maintainers are working on, they are splitting out Spherical Harmonics into a separate API. The thought process here is that in the future there may be no need for Spherical Harmonics, and they'd have another API they'd define later with those properties. Within our team here at Cesium we went back and forth on this trying to decide if we'd follow suit and we've decided against doing so.

Currently, the Spherical Harmonic data in KHR_gaussian_splatting is optional. Ideally, when PBR materials for 3D Gaussian splatting become possible in a wide variety of renderers, we will use the existing mechanisms for materials within glTF and the Spherical Harmonic data will simply be omitted. If there are special properties for those materials, a material extension can be defined later to provide those on the material object. Given the simplicity of this approach, we don't currently see a need within glTF for splitting Spherical Harmonics out into a separate extension.

[jo-chemla]

Thanks for the updates @weegeekps and taking inspiration from OpenUSD gsplats for aligning formalization discussion.

@willeastcott & @slimbuck Following the impressive work you handled around Self-Organizing Gaussians for rendering and conversion via playcanvas/splat-transform, would @playcanvas be interested in formalizing a glTF extension like KHR_sog_gaussian_splats_compression extending KHR_gaussian_splatting?

This could help formalize a standard specification for storing, consuming SOG-compressed gsplats for broader adoption - for example also within other renderers.

[javagl]

This could help formalize a standard specification for storing, consuming SOG-compressed gsplats for broader adoption

I think that this is the crux for even trying to extract a "baseline" specification like KHR_gaussian_splatting. I've been poking several people, inquiring whether the attributes that are currently defined in that baseline specification are suitable for something that could be called a "general splat support for glTF".

One could argue that such a baseline specification would be useless if KHR_spz_gaussian_splats_compression was the only specification that could be built on top of it. That's not entirely true, though: The baseline specification already does allow the splat data to be stored in good old plain accessors, in uncompressed form. And it already does allow to apply some of the existing compression machinery (like 'meshopt') on top of that. This means that there already is something that a renderer could rely on, and that producers could generate with relatively little effort, including a compressed form of splats.

But it could still be important to make sure that this structure (i.e. the attributes that are defined there) could be "filled" or "fed" with data that is compressed in other ways. If it is possible to fill that baseline spec with life, using both SPZ and SOG, then this is a strong hint that it could also be used for a third or fourth compression method that may be developed... tomorrow.

---

Caveats:

The use of that base extension:

As far as I understood, it is by no means clear how splat data is supposed to be sent to a renderer. Some renderers might try to keep it simple, and use VBOs (although I've heard that this may not be the best choice). Some shaders (e.g one that I took from a Python renderer and used in a Java renderer) are using SSBO (Shader Storage Buffer Objects). I don't know whether this is good or bad, but ... it was relatively simple, and it seems to work reasonably well. Others are taking the SPZ data and are encoding that into textures in some way. (It looks quirky, but ... usually, when something like that looks "quirky", people argue with "performance" (and usually, when people argue with "performance", there are no benchmarks... however)). I think that for SOGs, the data is already stored in textures to begin with, and it seems reasonable to assume that these textures are supposed to be sent to the shader directly.

So if someone wants to implement splat support for glTF in a renderer, then it should at least be possible to write a renderer that just consumes the attributes that are defined in the baseline specification. This may not always be the "best" choice, and it may have performance drawbacks for the rendering itself, or require forms of transcoding. But the renderer (or engine) still has the option to include deeper, more specific knowledge about the actual extension, and how the data is actually stored. The renderer could look for the SPZ version, and use that 'encoded-textures' approach. The renderer could look for the SOG version, and used these textures directly. If none of that is found, (maybe because the splats are just stored as plain accessors+meshopt, or another compression method), then it could still render them, and wouldn't have to bail out just because it doesn't know the latest-and-greatest "compressed splat rendering" method that someone came up with.

The limits of that base extension:

There will be animated splats. We know that. Will there be "morph targets" for splats? Maybe. Will there be additional attributes for splats? Probably, in some form. Can the additional information that is required for this "next-generation splats" be encoded in that baseline extension? No! And that is not the goal. When people agree on what "animated splats" are, and what attributes they need, then there can be a KHR_animated_gaussian_splatting extension that defines the baseline structure, and where different compression methods can fill that with life.

---

(All that may sound naive for someone who's more deeply involved in splat (rendering) research. I'm just trying to ensure that it makes sense to establish such a baseline specification. Feel free to chime in with additional thoughts about where this does or does not make sense)

[lexaknyazev]

3D Gaussian splats are effectively fields of 3D Gaussian splats

This sounds confusing. Could this statement be more clear?

[lexaknyazev]

Please name the attributes with the extension scope.

[weegeekps]

Just to confirm, we're fine with the current approach for namespacing? (i.e. KHR_gaussian_splatting:SCALE)

[lexaknyazev]

This scheme resolves the issue and there are no alternative proposals anyway.

[lexaknyazev]

Formats for these two attributes are defined in the base spec and POSITION is further extended by KHR_mesh_quantization.

This extension should not interfere with that.

[javagl]

Related to nianticlabs/spz#42 : It still has to be made clear whether these are the spherical harmonics coefficients, or actual colors.

[weegeekps]

I will remove POSITION and COLOR_0 from the definitions here, but are there concerns around us using these? How this extension will use them is identical to how they're defined in the base spec.

It still has to be made clear whether these are the spherical harmonics coefficients, or actual colors.

The intent here is that this is the actual the diffuse color combined with the opacity of the splat (alpha). We derive this by taking the diffuse color components of the Spherical Harmonic and multiplying them by the zeroth-order Spherical Harmonic coefficient. This is covered by the original article from INRIA.

[lexaknyazev]

The concern is to not redefine core semantics wrt their accessor formats. As currently written, the extension spec needlessly restricts them (if taken literally).

[javagl]

If these are omitted (since they are defined by the core spec), then it will imply that they are supposed to be the actual colors.

There is some potential for confusion with the range and way that SPZ stores this data, with the color values in [0, 255] being mapped to values in ~[-3.3, 3.3]. PLY is using the [-1.7, 1.7] range, and Gsplat is storing color bytes in [0, 255] and leaves the conversion into the first spherical harmonics coefficient range to the reader. But that may rather be discussed in the linked issue.

[lexaknyazev]

Alternatively, we may just drop COLOR_0 usage altogether and replace it with KHR_gaussian_splatting:TOTALLY_NOT_A_COLOR (name pending).

This would implicitly resolve several edge cases:

• COLOR_0 values are directly plugged into PBR's base color, which is not applicable to splats;
• COLOR_0 values must be clamped to 0...1, which may not always be convenient for splats;
• COLOR_0 values could have only three components with alpha being always opaque; this is likely not very useful for splats.

[javagl]

Nah, that name's fine. Let's ship it 😎

But more seriously: If this was considered, then following the other SH naming patterns and calling that SH_DEGREE_0_COEFFS could be totally fine. (These are the values referred to as f_dc_... in splat PLY files).

But I thought that the intention was exactly to make sure that this COLOR_0 should be THE glTF COLOR_0. The possibility of the "fallback to point cloud rendering" could be one reason. Having something that people already have agreed on is another. Referring to the bullet points, this would mean that people could use them like PBR colors. And they would be clamped to [0, 1].

Even if this was replaced by something like SH_DEGREE_0_COEFFS, this would still require us to define the exact value range and meaning. Some math-savvy splat expert could chime in here, and say whether these coefficients do have "strict" value ranges. I think that they do, insofar that a SH 0 coefficient of 100.0 simply ~"does not make sense", but I might be wrong.

Regarding the last bullet point: Note that there are similar (maybe even more tricky) questions for the specific case of the alpha component. Gaussian splats refer to that as "opacity", which oddly enough has a value range of [-Infinity, +Infinity]. Trying to map that to a "real" value range like [0,1] or even [0,255] has some caveats...

[jeanphilippepons]

Yes I agree with @javagl, the intention is exactly to make sure that this COLOR_0 should be THE glTF COLOR_0.
In this proposal, all the components of COLOR_0 including the alpha are already mapped from the raw gaussian splats attributes in the right way. In particular, the opacity range of [-Infinity, +Infinity] in the raw representation is just a sigmoid encoding. There is a well-defined mapping between this opacity and an alpha in [0,1] and it makes "physical" sense.

[javagl]

There is a well-defined mapping between this opacity and an alpha in [0,1] and it makes "physical" sense.

This "mapping" has some limits, particularly when the conversion to byte comes on top of that.

For reasons that I could hardly articulate, here's a table showing the "roundtrip":

• It contains the "input" alphaByte, from 0 to 255
• The alphaFloat is the result of converting that to float as alphaByte / 255.0
• The opacity is invSigmoid(alphaFloat)
• The alphaFloatResult is sigmoid(opacity)
• The alphaByteResult is (int)(alphaFloatResult * 255.0)

alphaByte|alphaFloat|   opacity|alphaFloatResult|alphaByteResult
---------+----------+----------+----------------+---------------
        0| 0.0000000| -Infinity|       0.0000000|              0
        1| 0.0039216|-5.5373344|       0.0039216|              0
        2| 0.0078431|-4.8402424|       0.0078431|              1
        3| 0.0117647|-4.4308167|       0.0117647|              3
        4| 0.0156863|-4.1391587|       0.0156863|              3
        5| 0.0196078|-3.9120231|       0.0196078|              4
        6| 0.0235294|-3.7256935|       0.0235294|              6
        7| 0.0274510|-3.5675185|       0.0274510|              7
        8| 0.0313726|-3.4299467|       0.0313726|              8
        9| 0.0352941|-3.3081069|       0.0352941|              9
       10| 0.0392157|-3.1986730|       0.0392157|             10
       11| 0.0431373|-3.0992730|       0.0431373|             11
       12| 0.0470588|-3.0081549|       0.0470588|             11
       13| 0.0509804|-2.9239883|       0.0509804|             13
       14| 0.0549020|-2.8457396|       0.0549020|             14
       15| 0.0588235|-2.7725887|       0.0588235|             15
       16| 0.0627451|-2.7038748|       0.0627451|             16
       17| 0.0666667|-2.6390572|       0.0666667|             17
       18| 0.0705882|-2.5776882|       0.0705882|             18
       19| 0.0745098|-2.5193927|       0.0745098|             19
       20| 0.0784314|-2.4638531|       0.0784314|             20
       21| 0.0823529|-2.4107985|       0.0823530|             21
       22| 0.0862745|-2.3599961|       0.0862745|             21
       23| 0.0901961|-2.3112431|       0.0901961|             23
       24| 0.0941176|-2.2643638|       0.0941177|             24
       25| 0.0980392|-2.2192035|       0.0980392|             25
       26| 0.1019608|-2.1756256|       0.1019608|             25
       27| 0.1058824|-2.1335087|       0.1058824|             27
       28| 0.1098039|-2.0927455|       0.1098039|             27
       29| 0.1137255|-2.0532391|       0.1137255|             29
       30| 0.1176471|-2.0149031|       0.1176471|             30
       31| 0.1215686|-1.9776589|       0.1215686|             30
       32| 0.1254902|-1.9414358|       0.1254902|             32
       33| 0.1294118|-1.9061698|       0.1294118|             33
       34| 0.1333333|-1.8718021|       0.1333333|             34
       35| 0.1372549|-1.8382795|       0.1372549|             35
       36| 0.1411765|-1.8055527|       0.1411765|             36
       37| 0.1450980|-1.7735771|       0.1450980|             37
       38| 0.1490196|-1.7423111|       0.1490196|             38
       39| 0.1529412|-1.7117168|       0.1529412|             38
       40| 0.1568628|-1.6817585|       0.1568628|             40
       41| 0.1607843|-1.6524040|       0.1607843|             41
       42| 0.1647059|-1.6236225|       0.1647059|             42
       43| 0.1686275|-1.5953861|       0.1686275|             43
       44| 0.1725490|-1.5676684|       0.1725490|             44
       45| 0.1764706|-1.5404450|       0.1764706|             45
       46| 0.1803922|-1.5136929|       0.1803921|             45
       47| 0.1843137|-1.4873904|       0.1843137|             47
       48| 0.1882353|-1.4615178|       0.1882353|             48
       49| 0.1921569|-1.4360559|       0.1921569|             49
       50| 0.1960784|-1.4109869|       0.1960784|             50
       51| 0.2000000|-1.3862944|       0.2000000|             51
       52| 0.2039216|-1.3619622|       0.2039216|             52
       53| 0.2078431|-1.3379757|       0.2078432|             53
       54| 0.2117647|-1.3143208|       0.2117647|             54
       55| 0.2156863|-1.2909842|       0.2156863|             55
       56| 0.2196078|-1.2679532|       0.2196078|             55
       57| 0.2235294|-1.2452158|       0.2235294|             57
       58| 0.2274510|-1.2227607|       0.2274510|             58
       59| 0.2313726|-1.2005773|       0.2313726|             59
       60| 0.2352941|-1.1786550|       0.2352941|             60
       61| 0.2392157|-1.1569843|       0.2392157|             61
       62| 0.2431373|-1.1355557|       0.2431373|             62
       63| 0.2470588|-1.1143607|       0.2470588|             63
       64| 0.2509804|-1.0933902|       0.2509804|             64
       65| 0.2549020|-1.0726367|       0.2549020|             65
       66| 0.2588235|-1.0520922|       0.2588235|             66
       67| 0.2627451|-1.0317492|       0.2627451|             67
       68| 0.2666667|-1.0116009|       0.2666667|             68
       69| 0.2705882|-0.9916401|       0.2705882|             69
       70| 0.2745098|-0.9718605|       0.2745098|             70
       71| 0.2784314|-0.9522558|       0.2784314|             71
       72| 0.2823530|-0.9328200|       0.2823530|             72
       73| 0.2862745|-0.9135472|       0.2862745|             73
       74| 0.2901961|-0.8944319|       0.2901961|             74
       75| 0.2941177|-0.8754687|       0.2941177|             75
       76| 0.2980392|-0.8566524|       0.2980392|             76
       77| 0.3019608|-0.8379781|       0.3019608|             77
       78| 0.3058824|-0.8194408|       0.3058824|             78
       79| 0.3098039|-0.8010361|       0.3098040|             79
       80| 0.3137255|-0.7827593|       0.3137255|             80
       81| 0.3176471|-0.7646061|       0.3176471|             81
       82| 0.3215686|-0.7465723|       0.3215686|             82
       83| 0.3254902|-0.7286538|       0.3254902|             82
       84| 0.3294118|-0.7108467|       0.3294118|             84
       85| 0.3333333|-0.6931471|       0.3333333|             85
       86| 0.3372549|-0.6755514|       0.3372549|             86
       87| 0.3411765|-0.6580558|       0.3411765|             86
       88| 0.3450980|-0.6406569|       0.3450981|             88
       89| 0.3490196|-0.6233514|       0.3490196|             89
       90| 0.3529412|-0.6061358|       0.3529412|             90
       91| 0.3568628|-0.5890069|       0.3568627|             90
       92| 0.3607843|-0.5719616|       0.3607843|             92
       93| 0.3647059|-0.5549968|       0.3647059|             93
       94| 0.3686275|-0.5381095|       0.3686275|             94
       95| 0.3725490|-0.5212969|       0.3725490|             95
       96| 0.3764706|-0.5045560|       0.3764706|             96
       97| 0.3803922|-0.4878840|       0.3803922|             97
       98| 0.3843137|-0.4712783|       0.3843137|             98
       99| 0.3882353|-0.4547361|       0.3882353|             99
      100| 0.3921569|-0.4382549|       0.3921569|            100
      101| 0.3960784|-0.4218321|       0.3960784|            101
      102| 0.4000000|-0.4054651|       0.4000000|            102
      103| 0.4039216|-0.3891515|       0.4039216|            103
      104| 0.4078431|-0.3728889|       0.4078432|            104
      105| 0.4117647|-0.3566749|       0.4117647|            105
      106| 0.4156863|-0.3405072|       0.4156863|            106
      107| 0.4196078|-0.3243834|       0.4196078|            107
      108| 0.4235294|-0.3083013|       0.4235294|            108
      109| 0.4274510|-0.2922587|       0.4274510|            109
      110| 0.4313726|-0.2762534|       0.4313726|            110
      111| 0.4352941|-0.2602831|       0.4352941|            111
      112| 0.4392157|-0.2443457|       0.4392157|            111
      113| 0.4431373|-0.2284392|       0.4431373|            113
      114| 0.4470588|-0.2125614|       0.4470588|            114
      115| 0.4509804|-0.1967103|       0.4509804|            115
      116| 0.4549020|-0.1808837|       0.4549020|            116
      117| 0.4588235|-0.1650797|       0.4588235|            117
      118| 0.4627451|-0.1492963|       0.4627451|            118
      119| 0.4666667|-0.1335314|       0.4666667|            119
      120| 0.4705882|-0.1177830|       0.4705882|            120
      121| 0.4745098|-0.1020492|       0.4745098|            120
      122| 0.4784314|-0.0863281|       0.4784314|            122
      123| 0.4823529|-0.0706176|       0.4823530|            123
      124| 0.4862745|-0.0549158|       0.4862745|            124
      125| 0.4901961|-0.0392207|       0.4901961|            125
      126| 0.4941176|-0.0235305|       0.4941177|            126
      127| 0.4980392|-0.0078432|       0.4980392|            127
      128| 0.5019608| 0.0078433|       0.5019608|            128
      129| 0.5058824| 0.0235306|       0.5058824|            129
      130| 0.5098040| 0.0392208|       0.5098040|            130
      131| 0.5137255| 0.0549159|       0.5137255|            131
      132| 0.5176471| 0.0706177|       0.5176471|            132
      133| 0.5215687| 0.0863282|       0.5215687|            133
      134| 0.5254902| 0.1020494|       0.5254902|            134
      135| 0.5294118| 0.1177832|       0.5294118|            135
      136| 0.5333334| 0.1335315|       0.5333334|            136
      137| 0.5372549| 0.1492964|       0.5372549|            137
      138| 0.5411765| 0.1650799|       0.5411765|            138
      139| 0.5450981| 0.1808839|       0.5450981|            139
      140| 0.5490196| 0.1967104|       0.5490196|            140
      141| 0.5529412| 0.2125615|       0.5529412|            141
      142| 0.5568628| 0.2284393|       0.5568628|            142
      143| 0.5607843| 0.2443459|       0.5607843|            143
      144| 0.5647059| 0.2602832|       0.5647059|            144
      145| 0.5686275| 0.2762535|       0.5686275|            145
      146| 0.5725490| 0.2922588|       0.5725490|            146
      147| 0.5764706| 0.3083014|       0.5764706|            147
      148| 0.5803922| 0.3243836|       0.5803922|            148
      149| 0.5843138| 0.3405073|       0.5843138|            149
      150| 0.5882353| 0.3566751|       0.5882353|            150
      151| 0.5921569| 0.3728890|       0.5921569|            151
      152| 0.5960785| 0.3891516|       0.5960785|            152
      153| 0.6000000| 0.4054652|       0.6000001|            153
      154| 0.6039216| 0.4218322|       0.6039216|            154
      155| 0.6078432| 0.4382550|       0.6078432|            155
      156| 0.6117647| 0.4547363|       0.6117647|            156
      157| 0.6156863| 0.4712784|       0.6156864|            157
      158| 0.6196079| 0.4878842|       0.6196079|            158
      159| 0.6235294| 0.5045561|       0.6235294|            159
      160| 0.6274510| 0.5212970|       0.6274510|            160
      161| 0.6313726| 0.5381097|       0.6313726|            161
      162| 0.6352941| 0.5549969|       0.6352941|            162
      163| 0.6392157| 0.5719617|       0.6392157|            163
      164| 0.6431373| 0.5890070|       0.6431373|            164
      165| 0.6470588| 0.6061359|       0.6470588|            165
      166| 0.6509804| 0.6233515|       0.6509804|            166
      167| 0.6549020| 0.6406571|       0.6549020|            167
      168| 0.6588235| 0.6580560|       0.6588235|            168
      169| 0.6627451| 0.6755515|       0.6627452|            169
      170| 0.6666667| 0.6931472|       0.6666667|            170
      171| 0.6705883| 0.7108468|       0.6705883|            171
      172| 0.6745098| 0.7286540|       0.6745098|            172
      173| 0.6784314| 0.7465724|       0.6784314|            173
      174| 0.6823530| 0.7646062|       0.6823530|            174
      175| 0.6862745| 0.7827594|       0.6862745|            175
      176| 0.6901961| 0.8010362|       0.6901961|            176
      177| 0.6941177| 0.8194410|       0.6941177|            177
      178| 0.6980392| 0.8379782|       0.6980392|            178
      179| 0.7019608| 0.8566526|       0.7019608|            179
      180| 0.7058824| 0.8754689|       0.7058824|            180
      181| 0.7098039| 0.8944320|       0.7098039|            181
      182| 0.7137255| 0.9135473|       0.7137255|            182
      183| 0.7176471| 0.9328201|       0.7176471|            183
      184| 0.7215686| 0.9522560|       0.7215686|            184
      185| 0.7254902| 0.9718606|       0.7254902|            185
      186| 0.7294118| 0.9916403|       0.7294118|            186
      187| 0.7333333| 1.0116010|       0.7333333|            187
      188| 0.7372549| 1.0317494|       0.7372549|            188
      189| 0.7411765| 1.0520923|       0.7411765|            189
      190| 0.7450981| 1.0726368|       0.7450981|            190
      191| 0.7490196| 1.0933905|       0.7490196|            191
      192| 0.7529412| 1.1143607|       0.7529412|            192
      193| 0.7568628| 1.1355559|       0.7568628|            193
      194| 0.7607843| 1.1569843|       0.7607843|            194
      195| 0.7647059| 1.1786550|       0.7647059|            195
      196| 0.7686275| 1.2005773|       0.7686275|            196
      197| 0.7725490| 1.2227608|       0.7725490|            197
      198| 0.7764706| 1.2452159|       0.7764706|            198
      199| 0.7803922| 1.2679532|       0.7803922|            199
      200| 0.7843137| 1.2909843|       0.7843137|            200
      201| 0.7882353| 1.3143209|       0.7882353|            201
      202| 0.7921569| 1.3379759|       0.7921569|            202
      203| 0.7960784| 1.3619623|       0.7960784|            203
      204| 0.8000000| 1.3862945|       0.8000000|            204
      205| 0.8039216| 1.4109870|       0.8039216|            205
      206| 0.8078431| 1.4360559|       0.8078431|            206
      207| 0.8117647| 1.4615178|       0.8117647|            207
      208| 0.8156863| 1.4873905|       0.8156863|            208
      209| 0.8196079| 1.5136930|       0.8196079|            209
      210| 0.8235294| 1.5404451|       0.8235294|            210
      211| 0.8274510| 1.5676686|       0.8274510|            211
      212| 0.8313726| 1.5953863|       0.8313726|            212
      213| 0.8352941| 1.6236227|       0.8352941|            212
      214| 0.8392157| 1.6524041|       0.8392157|            214
      215| 0.8431373| 1.6817586|       0.8431373|            215
      216| 0.8470588| 1.7117169|       0.8470588|            216
      217| 0.8509804| 1.7423112|       0.8509804|            217
      218| 0.8549020| 1.7735772|       0.8549020|            218
      219| 0.8588235| 1.8055528|       0.8588235|            219
      220| 0.8627451| 1.8382796|       0.8627451|            220
      221| 0.8666667| 1.8718022|       0.8666667|            221
      222| 0.8705882| 1.9061699|       0.8705882|            222
      223| 0.8745098| 1.9414359|       0.8745099|            223
      224| 0.8784314| 1.9776589|       0.8784314|            224
      225| 0.8823529| 2.0149031|       0.8823529|            225
      226| 0.8862745| 2.0532393|       0.8862745|            226
      227| 0.8901961| 2.0927455|       0.8901961|            227
      228| 0.8941177| 2.1335089|       0.8941177|            228
      229| 0.8980392| 2.1756256|       0.8980393|            229
      230| 0.9019608| 2.2192035|       0.9019608|            230
      231| 0.9058824| 2.2643640|       0.9058823|            230
      232| 0.9098039| 2.3112433|       0.9098039|            232
      233| 0.9137255| 2.3599961|       0.9137256|            233
      234| 0.9176471| 2.4107988|       0.9176471|            234
      235| 0.9215686| 2.4638534|       0.9215686|            235
      236| 0.9254902| 2.5193930|       0.9254902|            236
      237| 0.9294118| 2.5776885|       0.9294118|            237
      238| 0.9333333| 2.6390574|       0.9333333|            238
      239| 0.9372549| 2.7038748|       0.9372550|            239
      240| 0.9411765| 2.7725887|       0.9411765|            240
      241| 0.9450980| 2.8457396|       0.9450980|            241
      242| 0.9490196| 2.9239883|       0.9490196|            241
      243| 0.9529412| 3.0081549|       0.9529412|            243
      244| 0.9568627| 3.0992730|       0.9568627|            244
      245| 0.9607843| 3.1986732|       0.9607843|            245
      246| 0.9647059| 3.3081071|       0.9647059|            246
      247| 0.9686275| 3.4299469|       0.9686275|            247
      248| 0.9725490| 3.5675187|       0.9725490|            248
      249| 0.9764706| 3.7256935|       0.9764706|            249
      250| 0.9803922| 3.9120231|       0.9803922|            250
      251| 0.9843137| 4.1391587|       0.9843137|            251
      252| 0.9882353| 4.4308167|       0.9882354|            252
      253| 0.9921569| 4.8402424|       0.9921569|            253
      254| 0.9960784| 5.5373344|       0.9960784|            254
      255| 1.0000000|  Infinity|       1.0000000|            255

There are not really any surprises there. Each byte survives this "roundtrip". That "jump" from 5.53 to Infinity (for 0 and 255) is something to keep in mind, though. This refers to renderers, but also when trying to compute something like a "quantization error", or when there is a PLY file with opacity values like 6.0, 7.0, and 8.0 that - after a conversion to SPZ and back - all turn into Infinity...

[weegeekps]

Just to reinforce what @jeanphilippepons has already said, the intent is that COLOR_0 is the glTF-ready RGBA color with glTF-ready opacity. This is in fact so in the event that a renderer cannot render 3D Gaussian splats and no compression extension is being used, we can fall back to treating the points as a sparse point cloud.

Re: the Sigmoid encoding, there's not much I want to add here that hasn't already been said. If an implementor decides to map the data to the accessors rather than read the data directly from the SPZ blob, it must be stored as a 0.0 to 1.0 value within the alpha channel of COLOR_0 which means it needs to be post-Sigmoid decoding. So, while there may be round trip limitations from SPZ's standpoint, they don't impact the extension beyond being potentially lossy for some low values. Given SPZ is a lossy compression, and the impact is very minimal as demonstrated by the table above, this doesn't seem like an issue to me.

[lexaknyazev]

This entire section does not add anything to the normative part of the extension spec. It should either be moved to non-normative examples or removed entirely.

[lexaknyazev]

_INDEX is not a referable symbol in the spec, so it should not be named and formatted as one.

[weegeekps]

This is left over from a past iteration. Thanks for catching this. I need to go back through this entire implementation section again.

[lexaknyazev]

This requirement can be lifted. glTF accessors have a built-in fallback mechanism for such cases, see below.

[lexaknyazev]

See below wrt indices.

[lexaknyazev]

The accessor.bufferView property is defined as follows:

The index of the buffer view. When undefined, the accessor MUST be initialized with zeros; sparse property or extensions MAY override zeros with actual values.

This enables graceful fallback behavior for engines that do not support SPZ.

• If SPZ is supported, accessor data is sourced from the decompressed SPZ blob
• If SPZ is not supported and accessor.bufferView is defined, accessor data is sourced as usual.
• If SPZ is not supported and accessor.bufferView is undefined, there may be other extensions providing the data.

[javagl]

If SPZ is supported, accessor data is sourced from the decompressed SPZ blob

Related to the comment at #2490 (comment) : Is there anything that should (or has to be) said about the byteOffset of these accessors, depending on whether or not the bufferView is present?

(I think that the core(!) spec currently does not disallow a byteOffset, even when no bufferView is present. It's hard to imagine cases where this makes sense, but maybe there is no strong reason to disallow it either...?)

[lexaknyazev]

From the accessor.byteOffset definition:

This property MUST NOT be defined when bufferView is undefined.

[javagl]

I see, https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#_accessor_byteoffset (I only looked in the table for some reason)

[weegeekps]

I think graceful fallback is great, but I am struggling a bit to understand how I need to change this language to allow fallback. My understanding is that it's up to the runtimes to fill in the zeroes for an accessor without a bufferView property.

Given that, I don't get the impression that this statement is in violation of that? The intent here is to indicate that an attribute that is compressed by SPZ can't specify a bufferView (or a byteOffset, but that is probably not needed to be mentioned here).

I can see some clarity problems with the original statement. Would it be sufficient to include your bullet points as clarification?

[weegeekps]

Nevermind, I see the problem. I'll drop this statement and add some clarification about how fallback can work.

[lexaknyazev]

an attribute that is compressed by SPZ can't specify a bufferView

All such statements should be removed. The SPZ extension spec should just say that SPZ data takes priority over regular per-accessor buffer views.

[weegeekps]

What about attributes that aren't handled by SPZ? I'm primarily thinking of cases where someone may have a custom application attribute?

My gut is to specifically specify which attributes SPZ will provide data for, and then other attributes on a primitive are to be considered unhandled by SPZ. Is that alright?

[lexaknyazev]

The SPZ extension spec should list supported attributes and how to get them from SPZ data. I think it's safe to assume (but should still be mentioned) that other attributes are either sourced as usual or handled by additional extensions.

[weegeekps]

I added a section to show how the mapping between SPZ and the glTF accessors works. I also reworked the conformance section to satisfy your suggestions here. Please let me know if I need to make further changes in this regard.

[lexaknyazev]

There should be explicit language for matching attribute semantic names defined in KHR_gaussian_splatting to data structures defined in SPZ specs.

[lexaknyazev]

Both extension objects should be present here.

It might also be reasonable to put KHR_spz_gaussian_splats_compression inside to enforce extension dependency by design:

"extensions": {
  "KHR_gaussian_splatting": {
    "extensions": {
      "KHR_spz_gaussian_splats_compression": {
        "bufferView": 0
      }
    }
  }
}

[hybridherbst]

Isn't this a similar case as "KHR_texture_procedurals" and "EXT_texture_procedurals_mx_1_39"? Or do you see it differently?
https://github.com/KhronosGroup/glTF/blob/b13ced8e5dda85ad6b982b1b27bc5fe570306d38/extensions/2.0/Khronos/KHR_texture_procedurals/README.md#checkerboard-example

[lexaknyazev]

It's not the same. In procedural texturing extension, the base extension covers everything except the semantics of specific operations and thus it cannot be used on its own at all.

Here, the base extension is self-contained and the SPZ extension merely provides an alternative storage method.

[lexaknyazev]

Any particular reason why this name uses "splats" while the base extension uses "splatting"?

We should also put "spz" after "splats" for better sorting and alignment with other extension names.

[akmorgan99]

Would it make more sense to put the compression type at the end?
KHR_gaussian_splatting_compression_spz

We're doing this separately to leave other compression options open for the future. So for sorting and alignment, we put compression after splatting and then end with the specific compression type.

[javagl]

A strong +1 to prefixing all splat compression extension names with the base extension name KHR_gaussian_splatting....

[weegeekps]

Pushed some changes to the SPZ extension. Still working on updates to the base extension.

[weegeekps]

Pushed changes to the base 3DGS extension:

• Updated the extension based on feedback.
• Included the shape and rendering hints mentioned in my comment above.