Struct array extension

[Spiri0]

Related issue: #30394

This is a small extension of the new struct feature.
A storage can have millions of entries and in this storage there can not only be a single struct, but a huge amount of structs. These structs are then completely analogous to the many vec4 in a storage, only each struct contains a whole bundle of datas.

This is in my virtual geometry system the case where I have a lot of meshlets. Each meshlet is characterized by an extensive set of datas. However, the data sets of many meshlets can be bundled into just a few storages so that the large amount of data from the numerous meshlets of the GPU can be made available in a very compact manner in these few storages.

With this extension: The WGSLNodeBuilder recognizes from the length of the struct and the length of the storage array whether it is a single struct or a struct array

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	336.42 78.33	336.42 78.33	+0 B +0 B
WebGPU	518.01 143.8	518.2 143.84	+188 B +47 B
WebGPU Nodes	517.48 143.69	517.66 143.73	+188 B +46 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	465.41 112.17	465.41 112.17	+0 B +0 B
WebGPU	589.88 159.76	590.01 159.79	+122 B +26 B
WebGPU Nodes	545.25 149.23	545.37 149.26	+122 B +27 B

[sunag]

@Spiri0 Could you show me some use case of the API code in the issue? I'm not sure I understand the example part.

[Spiri0]

@Spiri0 Could you show me some use case of the API code in the issue? I'm not sure I understand the example part.

The simplest case will probably be where you combine position, normal, uv in an interleaved buffer, which is what I do in my virtual geometry system. Here is my somewhat simplified shader to see it better.

const interleavedStruct = struct( {
	position: 'vec3',
	normal: 'vec3',
	uv: 'vec2',
}, 'Interleaved' );

const vertexShader = wgslFn(`
 	fn main_vertex(
                projectionMatrix: mat4x4<f32>,
                cameraViewMatrix: mat4x4<f32>,
                vertexIndex: u32,
                instanceIndex: u32,
                maxTriangles: f32,
                instanceInfo: ptr<storage, array<u32>, read>,
                objectInfo: ptr<storage, array<ObjectInfo>, read>,
                meshWorldMatrix: ptr<storage, ${structName}, read>,
                interleavedBuffer: ptr<storage, array<Interleaved>, read>,
 	) -> vec4<f32> {

                var meshID = instanceInfo[ instanceIndex ];
                var object = objectInfo[ meshID ];
                var modelWorldMatrix = ${ modelWorldMatrix };

                var vertexId = vertexIndex + object.meshletID * 3u * u32( maxTriangles );

                var position = interleavedBuffer[ vertexId ].position;
                var normal = interleavedBuffer[ vertexId ].normal;
                var uv = interleavedBuffer[ vertexId ].uv;

                var outPosition = projectionMatrix * cameraViewMatrix * modelWorldMatrix * vec4f( position, 1 );

                varyings.meshletID = f32( object.meshletID );
                varyings.vNormal = normalize( normal );
                varyings.vUv = uv;

                return outPosition;
 	}

`, [ vMeshletID, vPosition, vNormal, vUv ]);

I no longer have a single classic attribute in my vertex shader. A compute shader decides which meshes should be drawn using drawIndirect and the ObjectInfo structArray data from the compute shader. I could have used 3 separate storages for position, normal, uv, which would then be vec3, vec3, vec2 arrays like you are used to. These 3 basic datas (position, normal, uv) can be easily combined in a structArray and sent to the GPU in a single storageBuffer.

So StructArrays are pretty analogous to the basic property arrays, vec2, vec3, vec4, ... but with them you can bundle data much more efficiently instead of having to use a single storage for each individual basic property.

In the picture on the right I have extended the WebGPU limits up to the maximum of my graphics card. So the maximum possible storageBuffer size that is possible. And yet the frame rate is still very high. Each colored area is a single mesh controlled with drawIndirect. This is basically an analog system to unreal's nanite system. You can't see a transition between the individual meshlets

The visibility of the many meshlets can only be controlled using structArrays. That's why, like you, I put so much energy into the struct extension, because drawIndirect relies on structArrays to be used really effectively. Together and with the storages and a compute shader, they overcome the limits with traditional attributes. With these tools in threejs, the mesh count is no longer the limiting factor.

In my app I also bundle all the camera datas that I need for drawIndirect in a single struct and storageBuffer and send them in each interval in one storage to the gpu. So I'm basically doing what @aardgoose intended to do with its expansion #27388
However, I do all the bundles by myself. I haven't yet considered about automated bundling. Bundling the camera datas for drawIndirect in a standard struct could certainly make sense for a standard visibility check on the gpu side. But we can think about it in peace. The most important thing is that everything necessary around drawIndirect is now together in threejs.
@Mugen87 I already planned to write to you that after this PR #28389 can be closed. The struct extension is an essential part of drawIndirect. It was a lot of work but it's worth it. No one can longer say threejs is not taking advantage of the potential of WebGPU :)

[sunag]

The Interleaved struct for StorageBuffer that you did is not an instance buffer, from what I understand you need the array to deal with the vertexId and struct? Can you share the code of your buffers too :)

[Spiri0]

Is that helpful ?
I create the interleaved with the meshoptimizer when loading the model. So these are already (position, norma. uv) of each meshlet in an array.

//the class method
createVerticesBuffer( meshlets ) {

     const vertexCount = meshlets.length * this.verticesPerMeshlet;
     const interleaved = new Float32Array( vertexCount * 12 );

     let interleavedIndex = 0;

     for ( let meshlet of meshlets ) {

         interleaved.set( meshlet.interleaved, interleavedIndex );

         interleavedIndex += meshlet.interleaved.length;

     }

     const interleavedBuffer = new THREE.StorageBufferAttribute( interleaved, 12 );

     return interleavedBuffer;
}


const interleavedStruct = struct( {
	position: 'vec3',
	normal: 'vec3',
	uv: 'vec2',
}, 'Interleaved' );

this.interleavedBuffer  = this.createVerticesBuffer( nonIndexedMeshlets );

const vertexShaderParams = {
    //all the other stuff
    interleavedBuffer: storage( this.interleavedBuffer, interleavedStruct, this.interleavedBuffer.count ).toReadOnly(),
}

I had also used 8 instead of 12 and placed uv.x in the w component of the position and uv.y in the w component of the normal vector, but that is exaggerated storage efficiency. So I pack the data of each vertex from all mesh fragments into a storage. For the dragon, in my current config there are over 1300 individual meshlets. The Interleaved storageBuffer therefore contains all the data that would normally be distributed across the three attributes (position, normal, uv).

In order to control which indexes belong to which mesh and which vertice in the mesh, several structArrays are necessary. One of my structs "meshletInfoStruct" is also a structArray and only appears in the compute shader like the DrawBuffer struct and is already a large struct with 48 elements per meshlet. With over 1300 meshlets that means over 1300 structs, each with 48 elements in a storage as an array. I've already toyed with the idea of ​​putting the whole project as a repository on Github. Data compression with structArrays is very powerful. Attribute-based geometry will be around for a long time. But I strongly assume that the storageBuffers, which allow free access to every element very efficiently with drawIndirect, are the future.

[sunag]

@Spiri0 Thanks for the example, what do you think about having structArray? A separate function for that.

[Spiri0]

Hi @sunag, I'm currently struggling with something as banal as the screenshot. Since the new chrome this doesn't work for me.
I'm completely open about a structArray. This would make it immediately clear from the node that it is an array of structs. I adjusted the example a little again. Now you can see that you can also bundle several datas in a buffer and decode them in the shader with the struct.
If you have something in mind with the structArrays, can you please also consider a method that can be used to read the number of structMembers? I mean that with the length which I added to the struct. This is very useful for determining buffer lengths when structs have no matrices but many (u/i/f, vec2,vec3,vec4) values. WebGPU then requires blocks of 4 for each member and then you can simply set the buffer length directly with the number of members * 4.

P.S. Thanks for the fix with the outputNode. It would have easily taken me a week to work my way through the code and find the issue.