Implemented THREE.InstancedMesh

[mrdoob]

First version of THREE.InstancedMesh:

var count = 10000;

var mesh = new THREE.InstancedMesh( geometry, material, count );

var dummy = new THREE.Object3D();

for ( var i = 0; i < count; i ++ ) {

	dummy.position.set(
		Math.random() * 20 - 10,
		Math.random() * 20 - 10,
		Math.random() * 20 - 10
	);

	dummy.rotation.set(
		Math.random() * Math.PI,
		Math.random() * Math.PI,
		Math.random() * Math.PI
	);

	dummy.updateMatrix();

	mesh.setMatrixAt( i, dummy.matrix );

}

scene.add( mesh );

Not super happy with the API yet... But it works 🙂

It's currently uploading 4 * 4 + 3 * 3 = 25 floats per instance.

If we uploaded position, quaternion and scale it would be 3 + 4 + 3 = 10 floats per instance but we'll have to run this per vertex.

Used #16141 and #16161 as reference, thanks @takahirox!

[looeee]

Nice!

The API look OK to me. Would these additions be possible?

var mesh = new THREE.InstancedMesh( geometry, material, count );

mesh.setPositionAt( i, x, y, z );
mesh.setScaleAt( i, x, y, z );
mesh.setRotationAt( i, x, y, z );

Or perhaps:

var v = new Vector3();
mesh.getPositionAt( i, v );

v.multiplyScalar( 2 );

mesh.setPositionAt( i, v );

[feiss]

This is great. 👍
Will the index of each instance be available in the vertex shader? I don't see where..

[spite]

Awesome!

A few questions:

• How are materials handled? Is every instance drawn with the same material properties?
• Can the count be modified? I imagine one can use .drawRange, but what about growing it beyond count, in case you need to dynamically shrink or expand the number of instances.
• Can the attributes be modified independently? i.e. upload only positions

For reference, this is the hacky solution i made for looper https://github.com/spite/looper/blob/master/modules/instanced.js
like this https://spite.github.io/looper/#199

[Mugen87]

More context: #5171

[EliasHasle]

Will the instance matrices not respect parent transforms?

[mrdoob]

@feiss

Will the index of each instance be available in the vertex shader? I don't see where..

I don't think the instancing API support that, but we could pass a integer attribute to the shader by default? What would you like to do with it?

[mrdoob]

@spite

How are materials handled? Is every instance drawn with the same material properties?

Yeah. Supporting all the material properties sounds hard, but maybe color per instance to start with?

Can the count be modified? I imagine one can use .drawRange, but what about growing it beyond count, in case you need to dynamically shrink or expand the number of instances.

I think abstracting shrinking should be easy. Expanding... not so much.

Can the attributes be modified independently? i.e. upload only positions

I want to do performance tests comparing passing matrices to the shader vs passing position/quaternion/scale?

Part of the reason we don't have a instancing abstraction yet is because every time we try we go down the rabbit hole of "using matrices is wasteful when I only want to modify the position".

[mrdoob]

@EliasHasle

Will the instance matrices not respect parent transforms?

It does already.

[takahirox]

Nice work! And I'm very glad to know that somehow I helped to get InstancedMesh into core.

I know it's very advanced one but are you interested in InstancedSkinnedMesh? Each instance can differently animate with 2d texture array.

Video / Demo

If you do, I'll share my idea when I have time. (Sorry I'm busy now!)

[mrdoob]

@takahirox InstancedSkinnedMesh 😁?

[takahirox]

Yes. Instancing even for SkinnedMesh. As the video and demo show, the humanoid models are renderered at a time with instancing but they differently animate, some of them run, some others of them walk, others don't move with different speed.

[mrdoob]

@takahirox Okay, lets leave that idea/discussion for a different PR, but let met know if the current design won't work well with what InstancedSkinnedMesh would need.

[takahirox]

@mrdoob Yes, I wasn't going to try to add SkinnedMesh support into this PR but I just wanted to share my experience for future improvement.

Will the index of each instance be available in the vertex shader? I don't see where..

I don't think the instancing API support that, but we could pass a integer attribute to the shader by default? What would you like to do with it?

IIRC, in WebGL2 gl_InstanceID is assigned for each instance but in WebGL 1 it isn't. So probably we need to manually pass integers as Ricardo mentioned in WebGL 1.

How are materials handled? Is every instance drawn with the same material properties?

Yeah. Supporting all the material properties sounds hard, but maybe color per instance to start with?

That sounds cool!

[mrdoob]

IIRC, in WebGL2 gl_InstanceID is assigned for each instance but in WebGL 1 it isn't. So probably we need to manually pass integers as Ricardo mentioned in WebGL 1.

Or maybe suggest people to use WebGL 2 for that use case 😛

[Usnul]

If we uploaded position, quaternion and scale it would be 3 + 4 + 3 = 10 floats per instance but we'll have to run this per vertex.

From my perspective per-vertex transform is preferable, personally. It's just a bunch of multiplications and additions, fast stuff. If instances have dynamic transforms - it takes the load of computing the matrix off the CPU, and added GPU cost is very low. In addition there's less memory load, so you might end up with similar performance due to lower memory footprint (i.e. better caching).

[Usnul]

Separate concern. I would prefer if there was a way to re-size number of instances. My current implementation allows that using fairly standard over-allocation and re-allocation approach.
When you need to grow - re-allocate memory to a larger chunk, when you shrink - just keep some unused elements within some reason until there are too many unused elements, then re-allocate. My current implementation is quite stable in this way with very few re-allocations happening as scene changes.

Here's my current API:
create():int - creates a new object and returns a reference to new object
remove(ref:int):boolean removes an object with a given reference, returns true if reference was removed, false if it was not found
setPosition(ref:int, x:float, y:float, z:float):void
setRotation(ref:int, x:float, y:float, z:float, w:float):void quaternion rotation
setScale(ref:int, x:float, y:float, z:float):void

[takahirox]

Some thoughts for instance matrix API. In #16161 I added .instances (Array<Object3D>) to InstancedMesh. Internally Three.js calculates instances' matrices from instances[x]'s position/quaternion(rotation)/scale and set to .instancedMatrix buffer attribute.

Pros is easy for users to control instance transform with familiar Object3D API, like mesh.instances[0].rotation.x += 0.01. Probably useful for Instancing + Animation like this demo I made in #16161

Cons is memory consumption overhead due to Object3D instances and performance overhead due to calculating matrices and copying to buffer attribute.

Even if this type of "easy to use API" won't get in core to avoid overhead, we may add helper under example.

[Usnul]

I think instancing is an advanced performance API, as such, I see 2 main issues to keep in mind:

1. Low per-instance memory overhead. You wouldn't use instancing if you have only a handful of objects
2. Low per-instance CPU overhead

If you are using instances - API doesn't have to be "comfortable", I believe. It should be as comfortable as it makes sense to do it, but not at the cost of performance attributes. These are my thoughts.

[takahirox]

BTW I want to make clear one thing. Material instance can't be shared between InstancedMesh and regular Mesh like it can't be between SkinnedMesh and regular Mesh because the shader will be specific to InstancedMesh, correct?

@Usnul Yeah I can understand and I know slow instancing doesn't make sense. So I lastly suggested adding helper under example (if needed) not doing in core. I suppose it may be useful for some users who aren't familiar with instancing or 3D, probably helper + instancing is still faster than without instancing.

[Mugen87]

Maybe it would be good to add a second demo that demonstrates raycasting with THREE.InstancedMesh. According to stackoverflow, the forum and my on-site support, it seems to me that apart from rendering instances, the interaction with single meshes is one of the most common tasks users want to do.

[pailhead]

Super long time ago there was this PR suggested, it's been working in the wild more or less like this since then, might be worth comparing notes:

#10750

[feiss]

Will the index of each instance be available in the vertex shader? I don't see where..

I don't think the instancing API support that, but we could pass a integer attribute to the shader by default? What would you like to do with it?

an integer would work. I'm thinking about setting different properties, like different colors or modify the vertices depending on the instance. You can set different properties using the position of the vertices, but that may be not enough, or doesn't work for some effect. Does it make sense?

[donmccurdy]

Are there other cases where reusing a single material on multiple objects would create multiple shader programs? Wondering if this should require a material.instancing flag or something, similar to .skinning.

[mrdoob]

Oh, I have not tested this.

[mrdoob]

I'll leave that for the next cycle.

[takahirox]

I've pointed out the similar stuff here, probably material can't be shared between regular mesh and instanced mesh.

And wondering if material can have multiple shader programs depending on some conditions, like SkinnedMesh, InstancedMesh, or so on. We can remove .skinning or such other properties from material and material can be shared between them.

Related #16356

[donmccurdy]

How are materials handled? Is every instance drawn with the same material properties?

Yeah. Supporting all the material properties sounds hard, but maybe color per instance to start with?

NodeMaterial actually provides a pretty decent instancing API. Modifying the transform of each instance is not as intuitive as the InstancedMesh API you've created here, but there's much more flexibility in terms of modifying any and all material properties:

const geometry = // ... create BufferGeometry using InstancedBufferAttribute ...

const material = new StandardNodeMaterial();

const instanceID = new AttributeNode( 'instanceID', 'float' );
const instanceCount = = new FloatNode( 100 );

// Set instance positions based on instance ID.
const rootPosition = new PositionNode();
material.position = new JoinNode(
  new SwitchNode( rootPosition, 'xy' ),
  new OperatorNode( new SwitchNode( rootPosition, 'z' ), instanceID, OperatorNode.MUL ) 
);

// Set per-instance material properties from attributes.
material.color = new AttributeNode( 'instanceColor', 'vec3' );
material.roughness = new AttributeNode( 'instanceRoughness', 'float' );

Example: https://three-shadenodeloader.donmccurdy.com

Not sure exactly what that means for this PR, except that it would be nice if the two techniques were compatible. If instance ID and instance count from InstancedMesh were available to NodeMaterial in some reliable way, that would be a good start. 🙂

[mrdoob]

I assume it uses InstancedBufferGeometry? I have not changed that code so it should still work.

https://github.com/mrdoob/three.js/blob/dev/src/renderers/WebGLRenderer.js#L866-L878

[donmccurdy]

Yep! I think it would still work, except that PositionNode probably doesn't know anything about the new instance position data yet.

[takahirox]

Out of curiosity. No InstancedMesh document yet. Does it mean it's still an experimental and for example its API can be changed? Or just lacking of the document?

[Mugen87]

Let's add the documentation and the TS declaration file in R110 👍

[takahirox]

Opened an issue as a reminder #17621

[hjlld]

the current frustum culling strategy will cut off the whole InstancedMesh.

[mrdoob]

@hjlld yes, that's known issue. InstancedMesh probably needs its own boundingSphere...

[EliasHasle]

But Mesh doesn't even have a bounding sphere, does it? boundingSphere and boundingBox are on geometry. I have been thinking that sometimes it should be possible to override the default behavior per object, as an alternative to disabling frustum culling for objects that are positioned in shaders.

[Usnul]

@mrdoob you might find it interesting to have a look at how I do culling for instanced meshed in meep
https://github.com/Usnul/meep/blob/a551bba1fe41748f895af9443f820c346ea49fd4/src/model/level/foliage/InstancedFoliage.js

I don't think that you can just copy my solution, since it has a fairly large foundation in spatial indices, object pools and other stuff. But it might give you some good ideas.

There are 2 parts to it, one is a compact representation for the instances, using a binary table with each row being a separate instance, second is a BVH for culling.

PS: Word "Foliage" in the name is purely historical, that's what I was creating the system for, it is entirely just an InstancedMesh clone in terms of function.

[hjlld]

I resolve this by following:

Because bounding info is totally on CPU side in my project, so i split them from geometries to an indivitual map. Then i set all InstancedMesh to frustumCulled = false, i also use BVH to make the culling strategy, so i put all the bounding infos to BVH. When BVH tells me the instance for this boudning info should be culled, i reconstruct the InstancedMesh, delete the corresponding matrix from InstancedMesh.instanceMatrix and make InstancedMesh.count --; in next frame, if BVH tells me the instance should be drawn, then i add the matrix and make count ++;

[EliasHasle]

@hjlld That sounds demanding for something to be done every frame.

[hookex]

When will r110 be released? very much look forward to, urgently need to use.

[Mugen87]

When will r110 be released?

October 30. But just to be clear, THREE.InstancedMesh can already be used with the current release R109: https://threejs.org/examples/webgl_instancing_suzanne

[hookex]

When will r110 be released?

October 30. But just to be clear, THREE.InstancedMesh can already be used with the current release R109: https://threejs.org/examples/webgl_instancing_suzanne

Wonderful job! Great guys!

[Mugen87]

FYI: I've used THREE.InstancedMesh in a new project and realized it makes sense to add the following line when creating the mesh if instanceMatrix is updated per frame like in this example.

mesh.instanceMatrix.setUsage( THREE.DynamicDrawUsage );

[nivashmani91]

Hi @mrdoob, Is there any possible ways to implement the instancing for lines and sprites?.

[Mugen87]

@nivashmani91 Please read #25078 for more information about that topic.